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Preface 


The  purpose  of  this  study  was  to  derive  a  method  for 
developing  departmental  measures  of  performance  in  a  white- 
collar  environment.  The  immediate  need  for  this  information 
is  to  assist  the  System  Program  Offices  (SPOs)  within 
Aeronautical  Systems  Division  in  the  departmental 
measurement  aspect  of  the  Total  Quality  Management  movement. 

An  integrated  measurement  development  approach  was 
compiled  through  extensive  literature  research.  This 
integrated  approach  was  then  grounded  with  interviews  with 
Pratt  &  Whitney's  Government  Engine  Business  to  provide  a 
practical  perspective  to  accompany  the  theoretical  concepts. 
The  final  step  in  this  process  would  be  to  test  the 
integrated  approach  within  one  or  more  SPOs  to  ascertain 
whether  this  approach  works  in  the  government  arena. 

This  thesis  effort  leaves  me  greatly  indebted  to  many 
people.  I  would  like  to  thank  my  advisor,  Lt  Col  John 
Shishoff  for  his  patience  and  support  throughout  the 
process.  I  am  also  grateful  to  Pratt  &  Whitney  for  their 
invaluable  assistance  and  participation.  I  would  like  to 
thank  my  family,  especially  my  brother,  Kevin,  for  being  an 
inspiration.  Finally,  I  would  like  to  thank  my  fiancee, 
Shanda,  for  helping  me  keep  everything  in  perspective  while 
providing  continued  understanding  and  thorough  support. 


Kirk  H.  Rumsey 
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Abstract 

, In  implementing  the  total  quality  initiative  taking  place 
within  Aeronautical  Systems  Division  of  Air  Force  Systems 
Command,  a  need  was  discovered  for  a  measurement  development 
procedure  designed  specifically  for  the  department  level  of 
a  white-collar  environment.  The  objective  of  this  research 
study  was  to  develop  a  procedure  to  fill  this  need. 

The  research  was  designed  around  answering  two  questions. 

'i  , 

The  first  question  was  what  do  the  authors  in  the  field  of 
measurement  say  is  the  best  way  to  develop  white-collar 
departmental  measures  of  performance?  This  question  was 
answered  by  reviewing  the  work  of  six  authors  and 
integrating  their  ideas  into  a  single,  integrated  procedure. 

The  second  question  was^how  did  several  Pratt  £•  Whitney 
Government  Engine  Business  departments  develop  their 
measures  of  performance?  This  answer  was  derived  by 
interviewing  twelve  managers  from  Pratt  &  Whitney. 

The  answers  to  the  above  questions  provided  an  integrated 
measurement  development  procedure, but  could  not  provide 
substantial  grounding  of  this  procedure  within  Pratt  & 
Whitney's  Government  Engine  Business.  The  procedure  is, 
however,  the  recommended  approach  for  developing 
departmental  measures  of  performance  within  a  white-collar 
environment . 
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DEVELOPING  DEPARTMENTAL  MEASURES  OF  PERFORMANCE 


IN  A  WHITE-COLLAR  ENVIRONMENT 


I.  Introduction 


1.1  General  Issue 

The  need  to  have  a  measurement  development  procedure 
designed  specifically  for  the  departmental  level  of  a  white- 
collar  environment  surfaced  as  a  result  of  the  Total  Quality 
Management  initiative  within  Air  Force  System  Command's 
Aeronautical  Systems  Division  (ASD) .  This  thesis  is 
designed  to  address  such  a  need. 

ASD  is  currently  participating  in  a  performance-enhancing 
Total  Quality  Management  (TQM)  program.  The  program 
consists  of  changing  the  philosophical  approaches  to  many  of 
the  business  practices  within  ASD.  Much  emphasis  is  being 
placed  on  customer  requirements  and  doing  things  right  the 
first  time  in  the  hope  of  improving  overall  performance. 

The  development  of  sound  departmental  performance  measures 
is  important  for  the  evaluation  of  progress  within  the  TQM 
program,  since  the  only  way  to  determine  improved 
performance  is  by  some  method  of  measurement. 

The  notion  of  improving  productivity  within  ASD  is  part 
of  a  larger  movement  found  throughout  industry.  Many 
companies  are  seeking  to  raise  their  performance  in  an 
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effort  to  upgrade  national  productivity  and  to  fight 
inflation,  as  well  as  to  remain  competitive  within  world 
markets.  Performance  measurement  provides  a  key,  along  with 
planned  managerial  undertakings,  to  improving  performance 
(Siegel,  1980:1). 

Developing  a  departmental  measurement  program  to  improve 
performance  includes  an  educational  aspect  of  how  to  teach  a 
department  to  do  measurement,  as  well  as  a  motivational 
aspect  of  how  to  inspire  a  department  to  do  measurement, 
finally,  it  also  includes  t.he  particular  problem  facing  ASD, 
which  is  how  to  measure  the  performance  of  its  departments 
(see  Figure  1,  page  5).  The  research  concentrated 
specifically  on  how  to  develop  measures  of  performance,  with 
no  attention  paid  to  the  motivational  or  educational  aspects 
of  the  larger  issue. 

1.2  Investigative  Questions 

The  research  effort  to  find  a  measurement  development 
procedure  for  a  white-collar  environment  led  to  two 
investigative  questions.  The  first  question  involved  what 
current  literature  is  available  on  the  subject  of 
departmental  measurement.  What  do  the  measurement  experts 
say  is  the  best  way  to  develop  white-collar  performance 
measures  at  the  department  level? 

The  second  investigative  question  involved  identifying 
what  a  private  sector  business  is  doing  in  this  field.  The 
private  sector  business  that  was  used  was  Pratt  &  Whitney. 
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Pratt  &  Whitney  was  chosen  because,  as  a  defense  contractor, 
many  of  its  departments  closely  resemble,  in  structure  and 
function,  the  departments  within  ASD.  Also,  Pratt  &  Whitney 
was  chosen  because  it  has  a  quality  program  in  place  and 
measurement  was  on  going  as  a  performance  evaluation  tool . 
With  potential  applicability  within  a  SPO  being  a  key  issue 
involving  the  results  of  the  research,  it  was  important  to 
choose  a  business  that  had  a  structure  similar  to  that  of 
the  SPO,  or  at  least  departments  within  the  business  that 
performed  functions  resembling  those  accomplished  in  a  SPO. 
How  does  this  business  measure  the  performance  of  its 
departments  and  how  were  these  measures  developed? 

1.3  Key  Definitions 

Prior  to  reviewing  the  research,  it  is  necessary  to 
understand  the  key  terms  used  throughout  the  thesis.  The 
key  terms  to  be  explained  are  measures  of  performance,  the 
associated  components  of  performance  measures,  and  white- 
collar  departments. 

According  to  Sink  and  Tuttle,  seven  criteria  must  be 
examined  when  measuring  performance  (Sink  and  Tuttle, 
1989:136).  These  measures  of  performance  are  evaluations  of 
how  well  a  department  is  doing  in  the  areas  of 
effectiveness,  efficiency,  quality,  productivity,  quality  of 
work  life,  innovation,  and  profitability/budgetability  (Sink 
and  Tuttle,  1989:136). 
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The  first  component  of  performance  measurement  is 
effectiveness.  Effectiveness  can  be  defined  as  the  degree 
to  which  an  organization  achieves  its  goals  (Daft  and 
Steers,  1986:334).  Paralleling  effectiveness  is  efficiency. 
These  two  concepts  are  closely  linked,  in  that  efficiency  is 
the  cost-benefit  ratio  incurred  in  the  pursuit  of 
organizational  goals  (Daft  and  Steers,  1986:335).  To 
illustrate  this  dual  concept,  an  army  could  be  effective  in 
winning  a  war,  but  its  measure  of  efficiency  would  not  be 
the  win,  but  the  number  of  soldiers  lost  or  the  amount  of 
time  spent  in  winning. 

Quality  is  the  third  criterion  evaluated  when  measuring 
performance.  According  to  the  Office  of  Management  and 
Budget  (OMB),  quality  is  the  extent  to  which  a  product  or 
service  conforms  to  requirements  and  meets  customer's  needs 
(OMB,  1989:3).  The  fourth  criterion  is  productivity,  which 
is  the  relationship  between  use  of  resources  and  the  results 
of  that  use  (Lehrer,  1983:31).  Quality  of  work  life  is 
number  five.  "Quality  of  work  life  means  that  one  receives 
psychic  personal  satisfaction  from  work  experience  in 
addition  to  economic  benefit"  (Lehrer,  1983:304). 

Innovation,  which  is  the  introduction  of  something  new, 
according  to  Webster's  Ninth  New  Collegiate  Dictionary,  is 
the  sixth  criterion  evaluated  when  measuring  performance, 
and  profitability/  budgetability  is  number  seven. 
Profitability  and  budgetability  refers  to  the  financial  gain 
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COMPONENTS  OF  A 
MEASUREMENT  PROGRAM 


EFFICIENCY 
QUAL ITY 
PRODUCTIVITY 
QUALITY  OF  Y.ORK  LIFE 
I  NKJOVAT  I  ON 
PRODUCTIVITY/ 
BUDGETAB IL ITY 


Figure  1:  Components  of  a  Measurement  Program 
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or  the  adherence  to  the  budget  that  the  target  system 
maintains . 

Sink  and  Tuttle  did  not  speak  of  a  hierarchy  of  the 
components  of  performance.  The  order  of  precedence,  then, 
depends  on  the  priorities  of  the  department.  In  fact,  many 
of  the  facets  of  performance  are  closely  related.  For 
example,  increases  in  productivity  generally  lead  to 
positive  effects  on  profitability,  sales,  capital, 
materials,  people,  competitiveness,  and  cost  per  unit  (Sink, 
1985:8) . 

For  this  thesis,  white-collar  departments  are  defined  as 

those  that  produce  no  hardware  product.  Department,  work 

environment,  target  system,  and  organization  are  all  terms 

which  will  be  used  throughout  the  thesis  to  refer  to  the 

actual  subject  of  the  measurement  system,  or  the  group  being 

measured.  For  the  purpose  of  this  thesis,  this  subject  is 

any  group  of  employees  within  a  business  entity  that  has  a 

unique  mission;  for  example,  a  contracting  group  for  a 

particular  engine  procurement  team,  or  a  specific  program 

management  team.  Other  examples  are  those  which  served  as 

pilot,  or  test,  organizations  for  a  recent  American 

Productivity  Center  two-year  study  in  the  area  of 

performance  measurement: 

accounting  and  finance,  customer  service, 
engineering,  facilities  management,  marketing 
and  sales,  operations,  information  systems 
management,  human  resources,  and  research  and 
development.  (Leth  and  others,  1985:3) 

The  study  had  thirteen  sponsors  including  General  Dynamics 
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Corporation,  McDonnell  Douglas  Corporation,  Johnson  & 
Johnson,  and  Rockwell  International  Corporation  as  well  as 
more  than  250  corporate,  labor,  government  and  academic 
supporters  (Leth  and  others,  1985:2-3). 

1.4  Scope  of  the  Research 

The  research  was  confined  to  investigating  procedures  for 
developing  measures  of  white-collar  department  performance, 
as  opposed  to  investigating  methods  of  developing  personal 
measures  of  performance.  As  alluded  to  previously  and  to  be 
discussed  in  more  depth  in  the  next  chapter,  departmental 
performance  measurement  is  the  area  of  measurement  causing 
difficulty  within  ASD  and  industry.  The  basis  for  the 
integrated  measurement  development  approach  was  the 
information  gathered  from  current  literature  on  the  subject 
of  departmental  performance  measurement.  The  interviews 
grounded  the  theoretical  baseline  derived  from  the 
literature  providing  the  practical  insight  associated  with 
the  managers'  perceptions  of  the  process  of  measurement 
development.  The  measures  discussed  with  the  managers 
during  the  interviews  also  serve  as  examples  for  departments 
starting  a  measurement  development  process. 

To  reiterate,  a  measurement  program  consists  of 
performance  measures,  motivation,  and  education.  The 
research  concentrated  specifically  of  developing  measures  of 
performance.  Addressing  the  issues  of  motivation  and 
education  is  beyond  the  scope  of  this  project. 


7 


1.5  Roadmap  to  the  Thesis 


The  thesis  follows  an  overall  research  theme  of 
developing  departmental  measures  of  performance  in  a  white- 
collar  environment.  Chapter  2  begins  with  a  discussion  of 
the  history  of  performance  measurement.  Also  in  Chapter  2 
is  an  analysis  of  the  problems  associated  with  white-collar 
performance  measurement  and  the  guidelines  to  be  followed 
when  creating  performance  measures.  This  chapter  serves  to 
set  a  baseline  understanding  of  the  problem  of  developing 
measures  of  performance. 

Chapter  3  is  the  methodology  used  for  the  research 
effort.  It  contains  an  explanation  of  how  the  literature 
analysis  was  conducted  and  what  sources  were  reviewed.  It 
also  contains  an  explanation  of  the  approach  to  the 
interviews  and  a  justification  of  why  case  study  interviews 
were  the  best  research  method  for  needed  information. 

Chapter  3  provides  the  logical  rationalization  for  the 
research  approach. 

Chapters  4  and  5  are  the  results  of  the  research. 

Chapter  4  is  the  analysis  of  literature  which  summarizes  the 
literature  review  and  integrates  the  information  into  a 
single  measurement  development  procedure.  Chapter  5 
compares  the  results  of  the  analysis  of  literature  with  the 
information  gathered  from  the  interviews. 

Chapter  6  contains  the  final  recommendations  and 
conclusions  that  can  be  made  from  analyzing  Chapters  4  and 
5.  The  recommendations  will  be  given  to  ASD  as  a 
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substantiated  approach  for  developing  measures  of 
performance.  Chapter  6  also  contains  the  recommendations 
for  further  study  which,  when  accomplished,  will  add  more 
valuable  research  conclusions  to  the  subject  of  developing 
departmental  measures  of  performance  in  a  white-collar 
environment . 
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Background  Literature 


2.1  Introduction  to  the  Chapter 

The  background  literature  is  designed  to  provide  the 
reader  with  an  historical  perspective  of  the  problem  of 
developing  departmental  measures  of  performance  in  a  white- 
collar  environment.  Also  included  in  this  chapter  is  a 
discussion  of  performance  measurement  problems  in  general, 
as  well  as  a  justification  for  conducting  performance 
measurement.  Finally,  the  chapter  provides  some  general 
guidelines  of  a  measurement  system.  When  developing 
measures  of  performance,  it  is  useful  to  know  prior  to 
starting  what  some  of  the  potential  pitfalls  might  be  and 
how  to  avoid  them. 

2.2  History  of  Performance  Measurement 

A  review  of  the  literature  produced  the  following  history 
of  performance  measurement.  Applicable  literature  generally 
concentrated  on  productivity  measurement.  For  example,  C.D. 
Wright,  the  first  U.S.  Commissioner  of  Labor  [Statistics] 
suggested  in  his  Sixth  Annual  Report  (1891)  that  many 
companies  were  in  a  position  to  compile  figures  on  their  own 
productivity  (Siegel,  1980:10).  His  report  was  based  on  a 
study  of  the  iron  and  steel  industry,  not  white-collar 
environments,  but  it  serves  to  illustrate  the  concern  for 
productivity  even  in  the  late-1800s.  The  Bureau  of  Labor 
Statistics  began  publishing  industry  productivity  indexes  in 
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1926  (Siegel,  1980:11).  However,  shortly  after  World  War 
II,  a  National  Industrial  Conference  Board  questionnaire 
revealed  that  while  many  companies  were  interested  in  labor 
productivity,  few  plants  had  quantitative  information  on  the 
subject  (Siegel,  1980:11).  Apparently,  the  fact  that  few 
plants  had  quantitative  productivity  information  remained 
unchanged  until  the  mid-60s.  At  that  point  in  time, 
businesses  became  even  more  interested  in  productivity. 
Irving  H.  Siegel,  author  of  Company  Productivity: 

Measurement  for  Improvement,  stated,  "American  businessmen 
have  been  obliged  by  economic  changes  since  the  mid-1960s  to 
become  more  explicitly  concerned  with  company  productivity 
improvement"  (Siegel,  1980:9).  The  economic  change  referred 
to  was  caused  by  the  persistence  of  unprecedented  rates  of 
inflation,  the  revolution  in  world  prices  of  petroleum  and 
in  conditions  of  supply,  the  proliferation  of  government- 
sponsored  regulation  and  paperwork,  and  the  apparent 
deterioration  of  employee  attitudes  (Siegel,  1980:9-10). 
Siegel  went  on  to  say,  "the  current  setting  favors  discovery 
and  adoption  by  many  more  companies  of  productivity 
monitoring  as  a  low-cost  means  of  spurring  productivity 
gain"  (Siegel,  1980:10).  Yet  even  now,  only  a  very  tiny 
fraction  of  the  firms  that  might  find  measurement  a  sound 
investment  seem  to  have  formal  systems  in  place  (Siegel, 
1980:10-11) . 
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2.3  Performance  Measurement  Problems  in  General 


As  noted  in  the  previous  section,  white-collar 

performance  measurement  has  met  with  its  share  of 

difficulty.  There  are  two  reasons  for  the  problems.  The 

first  is  the  vagueness  of  the  white-collar  business.  The 

second  is  the  stigma  surrounding  measurement. 

The  difficulties  in  developing  meaningful 
measures  of  performance  for  white-collar 
workers  are  well  known.  First,  most  white- 
collar  work  produces  no  tangible  end  product. 

Second,  even  if  input  and  output  could  be 
defined,  most  white-collar  workers  resist 
measurements.  (Boyett  and  Conn,  1988:209) 

One  of  the  inherent  characteristics  of  white-collar  work 
is  its  ambiguity  of  direction  and  product.  "The  mission  and 
objectives  of  most  white-collar  groups  are  vague.  Until  the 
vagueness  can  be  eliminated,  measures  cannot  be  developed" 
(Boyett  and  Conn,  1988:210).  This  vagueness  is  not  only  an 
external  perception,  but  also  an  internal  one.  Those  who 
conduct  analysis  of  white-collar  work,  such  as  Joseph  H. 
Boyett  and  Henry  P.  Conn,  authors  of  the  article  "Developing 
White-Collar  Performance  Measures,"  as  well  as  those  deeply 
rooted  in  the  system  have  identified  this  particular 
weakness . 

White  collar  workers  often  find  that  they  are 
providing  an  output  that  is  not  easy  to 
quantify.  Work  is  often  non- repetitive ,  and 
therefore,  by  definition,  a  standard  for 
doing  it  has  not  been  established.  (Lehrer, 

1983:38) 
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Because  of  the  vagueness  of  the  business,  many  mistakes 

can  be  found  in  the  practice  of  its  measurement.  These 

errors  fall  into  a  number  of  categories. 

What  we  find  in  practice  is  that  managers  at 
all  levels  and  in  all  kinds  of  organizations 
fall  into  traps  of: 

a)  Measuring  A  while  hoping  for  B.  We 
measure  the  easy  things,  the  most  pressing 
things,  the  wrong  things;  we  hope  for  quality 
while  measuring  and  controlling  only 
production  schedules. 

b)  Measuring  to  control  in  such  a  way  as  to 
make  improvement  more  difficult.  We  focus  on 
control  of  excess,  creating  a  compliance 
mentality  rather  than  an  improvement 
orientation . 

c)  Measuring  to  find  those  who  have 
performed  poorly  in  order  to  punish  them 
while  ignoring  the  good  performers.  (Sink  and 
Tuttle,  1989:1-2) 

The  inability  to  measure  performance  produces  a  negative 
stigma  surrounding  the  idea  of  measurement.  The  traps  that 
many  organizations  have  fallen  into  in  the  measurement  area 
have  led  to  employees  and  supervisors  alike  forming  negative 
paradigms  about  measurement.  Some  of  these  paradigms  and 
their  explanations  can  be  found  in  Table  1. 


2.4  Justification  for  Conducting  Performance  Measurement 
The  problems  associated  with  measurement  do  not  decrease 
the  need  to  measure.  From  Sink  and  Tuttle's  Planning  and 
Measurement  in  Your  Organization  of  the  Future,  the  overall 
justification,  for  conducting  measurement  in  any  environment 
starts  with  the  idea  of  management.  "The  essence  of 
management  is  that  one  cannot  manage  what  one  cannot 
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TABLE  1 


PARADIGMS  OF  MEASUREMENT 

(Sink  and  Tuttle,  1989:58-60) 

1.  Measurement  is  threatening.  The  primary  use  for  a 
measurement  system  is  for  punishment. 

2.  Precision  is  essential  to  useful  measurement.  Until  a 
completely  precise  measurement  system  is  developed,  there  is 
little  use  for  measurement. 

3.  Single  indicator  focus.  As  Robert  N.  Lehrer,  author  of 

White  Collar  Productivity,  says, 

...productivity  measurement  specialists 
sometimes  prescribe  very  detailed  and 
painstaking  programs  to  develop  the  exactly 
appropriate  measurement  system  in  an 
organization  which  must  be  completed  before 
productivity  improvement  is  allowed  to  begin. 
(Lehrer,  1983:29) 

Lehrer  continues  by  saying  that  it  is  quite  reasonable  to 
judge  productivity  on  a  collection  of  measures  (Lehrer, 
1983:41) . 

4.  There  must  be  an  emphasis  on  labor  productivity . 

Tangible  measures  have  become  an  obsession. 

5.  Subjective  measures  are  sloppy.  Many  managers  are 
uneasy  with  qualitative  measures,  and  without  hard  numbers 
they  feel  the  measures  are  of  no  value. 

6.  Standards  operate  as  ceilings  on  performance.  With 
goals  consisting  merely  of  standards,  that  becomes  the  limit 
to  departmental  performance. 
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measure"  (Sink  and  Tuttle,  1989:1).  They  go  on  to 

describe  measurement  as  a  tool  for  accomplishing  the 

performance  goals  of  the  manager. 

The  primary  goals  of  managers  in  all  areas 
and  at  all  levels  in  the  organization  should 
be  twofold: 

1.  Perform,  get  the  job  done;  and 

2.  Continuously  strive  to  improve 
performance.  (Sink  and  Tuttle,  1989:1) 

Thus,  according  to  Sink  and  Tuttle,  the  first  reason  to 
measure  is  to  use  measurement  as  a  performance  indicator. 
Robert  N.  Lehrer,  author  of  White  Collar  Productivity, 
speaks  of  the  need  for  measurement  to  "direct  attention  to 
parts  of  the  organization  that  are  in  a  stronger  or  weaker 
position  for  productivity  analysis  and  improvement"  (Lehrer 
1983:30)  . 

Sink  and  Tuttle's  second  reason  to  measure  is  to  improve 

Both  the  OMB  and  Robert  Lehrer  «urj2rt  this  concept.  Lehre 

states  that,  "A  measurement  system  is  a  tool  to  direct 

scarce  resources  to  the  targets  where  the  most  benefit  can 

be  obtained  from  those  scarce  resources"  (Lehrer,  1983:29). 

The  OMB  speaks  highly  of  measuring  to  improve.  "One 

critical  element  of  managing  for  continuous  improvement  is 

to  know  the  level  of  quality  being  achieved  at  any  given 

time  and  this  requires  the  use  of  quality  measures"  (OMB, 

1989:3).  The  OMB  goes  on  to  say  that  "measures  support 

improvement"  (OMB,  1989:3). 

Why  measure?  Measure  to  improve.  Measure  to 
provide  your  management  team  with  new 
insights  into  why  the  system  performs  the  way 
it  does,  where  it  can  be  improved,  and  when 
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the  system  is  in  control  or  out  of  control 
(Sink  and  Tuttle,  1989:1). 


2.5  General  Guidelines  of  a  Measurement  System. 

In  order  to  provide  performance  indicators  that  enhance 
performance  within  any  white-collar  environment,  there  are 
certain  principles  that  should  be  followed.  To  begin  with. 
Sink  and  Tuttle  discuss  several  principles  that  support  the 
theory  and  practice  of  measurement.  These  can  be  found  in 
Table  2. 

Along  with  following  the  guidelines  listed  and  explained 

in  Table  2,  the  method  chosen  for  conducting  departmental 

measurement  should  be  flexible;  one  that  can  be  adapted  to 

the  organization  at  hand. 

The  method  that  is  ultimately  selected  for 
use  will  depend  on  several  factors: 
applicability  to  type  of  work  process, 
presence  or  absence  of  checklists/cost  data, 
ease  of  use,  etc....  (OMB,  1989:23) 

With  flexibility  should  come  appropriateness. 

...the  output  and  the  input  must  be  from  the 
same  general  process.  It  is  not  very 
meaningful  to  talk  about  a  productivity 
measure  of  tons  of  steel  per  typist  employed. 
(Lehrer,  1983:31) 

The  measures  developed  should  be  congruent  with  the  goals 
of  the  organization.  This  is  avoiding  the  "measuring  A 
while  hoping  for  B"  trap.  Also,  the  measures,  should  not 
reward  an  employee  for  performing  in  a  manner  contrary  to 
the  good  of  the  company.  W.  Edwards  Demming,  the  man 
recognized  for  revitalizing  Japanese  industry,  offers  a  word 
of  advice  in  this  area.  "Quotas  take  account  only  of 


16 


TABLE  2 


GUIDELINES  OF  A  MEASUREMENT  SYSTEM 

(Sink  and  Tuttle,  1989:211-213) 


1.  Measurement  cannot  be  used  to  drive  performance 
improvement  -  the  driver  must  be  the  business  strategy  and 
the  performance  improvement  plan. 

2.  Acceptance  of  the  measurement  process  is  essential  to 
its  success  as  a  performance  improvement  tool . 

3.  Measure  what's  important  -  not  what's  easy  to  measure. 

4.  Adopt  an  experimental  approach  to  measurement  systems 
for  improvement. 

5.  Eliminate  the  use  of  numerical  goals,  work  standards, 
and  quotas. 

6.  What  is  needed  is  not  a  set  of  measurements  created  by 
experts  and  imposed  on  organizations,  but  rather  a  method  by 
which  management  teams  and  their  various  clientele  can 
create  performance  measurement  systems  suited  to  their  own 
inevitably  special  needs  and  circumstances. 

7.  The  greater  the  participation  in  the  process  in  creating 
a  performance  measurement  system,  the  greater  the  resulting 
performance  change,  and  the  greater  the  ease  of 
implementation  of  future  changes  based  upon  performance 
measurement . 

8.  Any  system  should  result  in  a  vector  of  performance 
measures,  not  attempting  to  achieve  a  single  measure.  Much 
of  the  controversy  and  lack  of  acceptance  stems  from 
attempts  to  make  a  very  complex  problem  appear  simple. 

9.  A  performance  measurement  must  not  appear  to  those 
involved  as  a  passing  fad. 

10.  A  performance  measurement  system  must  clearly  fit  into 
the  management  process  and  be  acknowledged  as  a  decision¬ 
making  and  problem-solving  support. 

11.  The  behavioral  consequences,  the  unintended  and 
potentially  dysfunctional  consequence  of  performance 
measurement  must  be  anticipated  and  reflected  in  system 
design . 

12.  A  useful  system  must  be  seen  by  those  whose  behaviors 
are  being  assessed  as  being  nonmanipul ati ve ,  not  gamed. 
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TABLE  2  (continued) 


13.  An  effective  measurement  system  must  build  upon 
consistent  and  well  understood  operation  definitions  for  the 
seven  performance  criteria. 

14.  The  unit  of  anal ysis/ target  system  must  be  clearly 
defined  and  delineated  in  order  for  measurement  to  succeed. 
An  input  output  analysis  is  a  necessary  precondition. 

15.  One  must  create  visibility  and  ownership  for  the 
resulting  measurement  systems  in  order  to  ensure  effective 
long-term  utilization. 

16.  One  must  clearly  separate  the  process  of  measurement 
from  the  process  of  evaluation.  The  difference  between  a 
control  chart  and  specifications/requirements  and  standards 
must  be  understood. 
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numbers,  not  quality  or  methods.  They  are  usually  a 
guarantee  of  inefficiency  and  high  cost.  A  person,  to  hold 
a  job,  meets  a  quota  at  any  cost,  without  regard  to  damage 
to  the  company"  (Walton,  1986:36). 


2.6  Conclusion 


The  bottom  line  is  that  measurement  is  vital  to  the 
success  of  the  organization  or  department.  It  fits  right  in 
to  Demming's  Fourteen  Points,  which  can  be  found  in  Table  3. 

Demming  talks  about  measuring  for  improvement,  not 

measuring  for  the  sake  of  measurement.  C.  Jackson  Grayson, 

author  of  the  article  "...But  what  do  I  do  Monday  Morning?", 

reiterates  the  importance  of  measurement  when  offering 

advice  for  improving  business. 

Create,  in  the  next  six  months,  an  improved 
measurement  system  that  focuses  on 
productivity  and  quality.  Disseminate  those 
measures  throughout  the  organization  and 
incorporate  them  into  your  budget, 
performance  appraisal  system,  compensation 
system,  and  reporting  system. . . . 

(Grayson: 239) 

In  summary,  John  Madden,  retired  National  Football  League 
coach  and  television  analyst,  simplified  the  rationale,  the 
whole  reason  for  conducting  measurement  in  the  first  place, 
when  he  said, 

I  just  wish  everyone  we  deal  with  in  the  real 
world  had  a  record  that  anybody  could  check. 

If  you  hire  a  lawyer  to  defend  you  in  an  auto 
accident,  do  you  have  somebody  who's  31-1  in 
auto  accident  cases  or  4-28?  Nobody  knows. 

(Madden,  1988:9) 
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TABLE  3 


DEMMING’S  FOURTEEN  POINTS 

(Walton,  1986:35-36) 

1.  Create  Constance  of  purpose  for  improvement  of  product 
and  service. 

2.  Adopt  the  new  philosophy. 

3.  Cease  dependence  on  mass  inspection. 

4.  End  the  practice  of  awarding  business  on  price  tag 
alone . 

5.  Improve  constantly  and  forever  the  system  of  production 
and  service. 

6.  Institute  training. 

7.  Institute  leadership. 

8.  Drive  out  fear. 

9.  Break  down  barriers  between  staff  areas. 

10.  Eliminate  slogans,  exhortations,  and  targets  for  the 
workforce . 

11.  Eliminate  numerical  quotas. 

12.  Remove  barriers  to  pride  of  workmanship. 

13.  Institute  a  vigorous  program  of  education  and 
retraining . 

14.  Take  action  to  accomplish  the  transformation. 
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III.  Methodology 


3.1  Approach 

The  methodology  was  designed  around  answering  two 
investigative  questions.  The  first  investigative  question 
was  what  do  the  writers  of  literature  in  the  field  of 
measurement  propose  as  the  best  way  to  develop  measures  of 
white-collar  departmental  performance?  The  answer,  found 
through  an  analysis  of  literature,  provided  an  integrated 
measurement  development  technique. 

The  second  investigative  question  was  how  does  a  selected 
business  measure  the  performance  of  its  departments,  and  how 
were  the  measures  developed?  The  answer  to  the  second 
question  was  found  by  conducting  a  case  study  of  Pratt  & 
Whitney's  Government  Engine  Division.  The  analysis  of  the 
second  question  either  supported  or  rejected  the  proposition 
that  the  integrated  measurement  technique  derived  from  the 
literature  can  be  supported  with  an  example  from  existing 
business  practices  in  an  environment  that  approximated  a 
System  Program  Office.  The  reason  for  starting  with  these 
two  questions  is  because  the  research  was  designed  to 
determine  a  theoretical  viewpoint,  ground  its  use  in 
practice,  and  recommend  future  action.  In  every  test  there 
must  be  a  standard  -  something  against  which  to  measure 
(Feldman,  1981:7). 

This  chapter  is  organized  in  the  same  manner  that  the 
research  was  conducted.  It  begins  with  the  Assessment  of 
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Literature  Section,  which  explains  the  approach  to  the 
analysis  of  literature  and  highlights  the  purpose  for 
conducting  the  analysis.  Also  contained  in  the  Assessment 
of  Literature  Section  are  the  background  of  the  key  sources 
and  a  discussion  of  how  the  literature  data  will  be 
presented  in  Chapter  4.  Following  the  Assessment  of 
Literature  Section  is  the  explanation  of  the  approach  to  the 
case  study.  The  explanation  begins  with  a  discussion  of  why 
case  study  interviews  were  conducted,  then  it  talks  about 
why  Pratt  &  Whitney  was  chosen  for  the  study,  and  finally, 
how  the  data  was  used  and  presented  in  Chapter  5.  The 
Methodology  concludes  with  a  discussion  of  validity  and  how 
validity  relates  to  this  research  effort. 

3.2  Assessment  of  Literature 

The  first  investigative  question  addressed  by  following 
the  methodology  was  what  do  the  writers  of  literature  in  the 
field  of  measurement  propose  as  the  best  way  to  develop 
measures  of  white-collar  departmental  performance?  The 
purpose  for  conducting  the  analysis  of  literature  was  to 
establish  a  theoretical  baseline  for  developing  measures  of 
performance.  The  answer  to  this  question  formed  the 
backbone  of  the  recommendations. 

This  question  was  answered  by  thoroughly  reviewing  the 
current  literature  on  this  subject.  The  search  through  the 
literature  was  not  only  for  performance  measurement 
development  techniques  but  also  for  techniques  for  measuring 
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any  of  the  seven  components  of  performance  discussed  in 
Chapter  1.  The  reason  for  expanding  the  search  was  to 
encounter  as  many  white-collar  measurement  development 
techniques  as  possible.  An  summary  of  the  results  of  this 
expansion  is  included  in  Chapter  6.  The  search  was  for  a 
noticeable  trend  among  the  recipes  dictated  by  the 
measurement  experts.  A  qualitative  trend  of  this  sort 
cannot  be  substantiated  statistically.  Nor  were  there  a 
number  of  complete  recurrences  that  will  confirm  a  trend. 
What  the  research  was  looking  for  was  a  common  thread  of 
ideas  presented  throughout  the  literature;  as  individual 
concepts  for  developing  measures  or  as  completed  recipes, 
that,  when  combined,  produced  a  defensible  measurement 
development  procedure. 

3.2.1  Background  of  the  Key  Sources.  The  authors  of  si 
sample  measurement  development  procedures  served  as  the 
primary  focus  of  summarization.  The  following  six 
paragraphs  contain  information  intended  to  provide  insight 
into  the  qualifications  of  each  of  the  authors. 

The  first  measurement  development  procedure  cited  in  the 
text  comes  from  Productivity  and  Quality:  Measurement  as  a 
Basis  for  Improvement,  by  Everett  E.  Adam,  James  C. 
Hershauer,  and  William  C.  Ruch.  The  book  stated  that  the 
authors’  academic  backgrounds  are  from  the  University  of 
Missouri  -  Columbia  and  Arizona  State  University.  Their 
research  includes  work  with  the  First  National  Bank  of 
Mobile,  Alabama,  General  Telephone  and  Electronics  (GTE), 
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and  the  Marriott  lodging-food  services  firm.  The  authors 
have  written  articles  in  the  field  of  performance  for  the 
Journal  of  Applied  Psychology,  The  Academy  of  Management 
Journal ,  the  American  Society  for  Quality  Control  Technical 
Conference  Transactions,  and  the  Academy  of  Management 
Review ,  to  name  but  a  few. 

The  second  measurement  development  procedure  cited  in  the 
text  comes  from  Planning  and  Measurement  in  Your 
Organization  of  the  Future,  by  D.  Scott  Sink  and  Thomas  C. 
Tuttle.  The  authors'  experiences  comes  from  both  the 
academic  environment  as  well  industry  and  public  service. 
They  integrated  what  they  learned  in  the  previous  twelve 
years  from  managing  quality,  productivity,  and  quality  of 
work  life  research  and  development  centers  at  major 
universities.  Sink  was  at  Ohio  State  University,  Oklahoma 
State  University,  and  Virginia  Tech,  while  Tuttle  was  at  the 
University  of  Maryland  (Sink  and  Tuttle,  1989:ix).  Their 
non-academic  experience  comes  from  the  organizations  with 
whom  they  have  worked.  These  organizations  include  the 
Navy,  the  Air  Force,  Baltimore  Gas  and  Electric,  San  Diego 
Gas  and  Electric,  Virginia  Fibre  Corporation,  Virginia 
Department  of  Transportation,  Norfolk  Naval  Shipyard,  Naval 
Sea  Support  Center  -  Atlantic,  Naval  Ordnance  Station  - 
Indian  Head,  Naval  Air  Depot  -  San  Diego,  Veteran's 
Administration,  Rhone  Poulence  (United  States  and  France), 
and  the  Internal  Revenue  Service  (Sink  and  Tuttle,  1989 :xi). 

The  third  measurement  development  procedure  cited  in  the 
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text  was  prepared  by  the  staff  of  the  Federal  Quality  and 
Productivity  Improvement  Program  in  the  Office  of  Management 
and  Budget.  Private  industries  such  as  Westinghouse 
Corporation,  3M,  Cigna  Health  Plans,  Honeywell  Aerospace  and 
Defense,  Xerox,  and  First  National  Bank  of  Chicago  assisted 
in  the  research  effort  by  providing  information  about  their 
measurement  systems. 

The  fourth  citation  containing  a  measurement  development 
procedure  is  from  the  Productivity  Measurement  Handbook,  by 
William  Christopher.  William  Christopher's  career  includes 
thirty  years  with  General  Electric  Company,  Hooker  Chemical 
Corporation,  an'4  'ccidental  Petroleum  Corporation  in 
marketing,  r r'  luct  and  venture  development,  internal 
consulting,  and  management.  He  is  the  president  and  senior 
management  counselor  of  The  Management  Innovations  Group, 
which  specializes  in  "Key  Performance  Area"  counsel  and 
implementation  services.  His  publications  include  two  books 
on  business  management:  The  Achieving  Enterprise,  Amacom, 
1974  (winner  of  the  1976  James  A.  Hamilton  Award),  and 
Management  for  the  1980s.  published  in  1980  by  Amacom 
(hardcover)  and  Prentice-Hall  (paperback)  (Christopher, 

1985: v) . 

The  fifth  measurement  development  procedure  cited  in  the 
text  is  from  Measuring  and  Assessing  Organizations,  by 
Andrew  H.  Van  Der  Ven  and  Diane  L.  Ferry.  The  information 
contained  in  the  procedure  was  tested  during  1975  and  1976 
in  twenty  California  employment  service,  unemployment 
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insurance,  and  vocational  rehabilitation  offices  consisting 
of  250  work  units  and  about  1300  organizations.  These  tests 
compared  favorably  to  the  tests  conducted  previously  at  the 
Wisconsin  Department  of  Industry,  Labor  and  Human  Relations 
(Van  Der  Ven  and  Ferry,  1980 : xi-xii ) . 

The  sixth  measurement  development  technique  cited  in  the 
text  is  from  Company  Productivity:  Measurement  for 
Improvement ,  by  Irving  H.  Siegel.  Mr.  Siegel  is  a 
consulting  economist,  who  received  his  Ph.D.  from  Columbia 
University.  His  Doctoral  dissertation,  Concepts  and 
Measurement  of  Production  and  Productivity,  republished  by 
the  Bureau  of  Labor  statistics  in  1952,  was  widely  used  by 
researchers  and  educators  in  many  countries  for  almost  a 
decade.  Mr.  Siegel  has  served  the  government  as  chief 
economist  of  the  Veteran's  Administration,  member  of  the 
economic  staff  of  the  President’s  Council  of  Economic 
Advisers,  and  economic  advisor  to  the  Bureau  of  Domestic 
Business  Development  in  the  Department  of  Commerce.  He  is  a 
fellow  of  the  American  Statistical  Association,  the  American 
Association  for  the  Advancement  of  Science,  and  the  New  York 
Academy  of  Sciences  (Siegel,  1980:v). 

3.2.2  Analysis  of  Sources.  Each  of  the  six  sources 
provided  a  measurement  development  recipe  to  be  followed  by 
a  white-collar  department  when  performance  measures  are  the 
desired  outcome.  By  comparing  the  recipes  to  each  other,  it 
was  possible  to  arrive  at  a  single,  integrated  approach  to 
developing  measures  of  performance.  Then,  in  tabular 
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format,  each  recipe  could  be  presented  next  to  the 
integrated  approach  to  determine  if  all  the  concepts  stated 
in  the  source  were  represented  by  the  integrated  approach. 
This  analysis  is  contained  in  Chapter  4. 

3.3  The  Choice  of  a  Case  Study 

Once  the  theoretical  baseline  had  been  established,  it 
was  necessary  to  attempt  to  ground  the  baseline  in  practice. 
This  was  done  by  answering  the  second  investigative 
question,  which  was  how  does  a  selected  business  measure  the 
performance  of  its  departments,  and  how  were  the  measures 
developed?  A  case  study  was  the  best  method  for  answering 
this  question. 

The  majority  of  information  cited  within  this  portion  of 
the  methodology  is  from  Robert  K.  Yin's  book  entitled  Case 
Study  Research:  Design  and  Methods.  Dr.  Yin’s  background 
includes  a  B.A.  (magna  cum  laude)  from  Harvard  College  and  a 
Ph.D.  from  the  Department  of  Psychology,  Massachusetts 
Institute  of  Technology.  Dr.  Yin  was,  at  the  time  of  the 
publication  of  his  book,  the  President  of  a  social  science 
research  firm.  He  also  served  as  Visiting  Professor  in  the 
Department  of  Urban  Studies  and  Planning  at  the 
Massachusetts  Institute  of  Technology.  His  research  is 
focussed  primarily  on  organizational  processes  (Yin, 
1984:160)  . 

The  case  study  is  but  one  of  several  ways 
of  doing  social  science  research.  Other  ways 
include  experiments,  surveys,  histories,  and 
the  analysis  of  archival  information  (as  in 
economic  studies).  Each  strategy  has 
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peculiar  advantages  and  disadvantages, 
depending  upon  three  conditions:  1)  the  type 
of  research  question,  2)  the  control  an 
investigator  has  over  actual  behavioral 
events,  and  3)  the  focus  on  contemporary  as 
opposed  to  historical  phenomena. 

In  general ,  case  studies  are  the  preferred 
strategy  when  "how"  or  "why"  questions  are 
being  posed,  when  the  investigator  has  little 
control  over  events,  and  when  the  focus  is  on 
a  contemporary  phenomenon  within  some  real- 
life  context.  Such  "explanatory"  case 
studies  also  can  be  complemented  by  two  other 
types  -  "exploratory"  and  "descriptive"  case 
studies.  (Yin,  1984:13) 

With  this  introduction  into  the  characteristics  of  a  case 
study,  the  choice  of  a  case  study  methodology  to  answer  the 
"how"  question  of  how  does  a  private  industry  develop 
departmental  measures  of  performance  was  obvious.  The 
proposition  itself  designates  the  overall  type  of  study  to 
be  performed.  The  proposition  being  investigated,  whether 
or  not  the  measurement  development  procedures  highlighted  in 
literature  are  applicable  in  practice,  addresses 
contemporary  events  best  observed  without  manipulating  the 
behaviors  of  those  being  observed.  The  nature  of  the 
material  indicates  that  a  case  study  is  appropriate.  Robert 
K.  Yin  goes  on  to  say  that  the  "case  study  is  preferred  in 
examining  contemporary  events,  but  when  the  relevant 
behaviors  cannot  be  manipulated"  (Yin,  1984:19). 

A  case  study  was  conducted  on  Pratt  &  Whitney's 
Government  Engine  Business  in  order  to  determine  if  the 
theoretical  measurement  development  procedures  were  being 
followed  in  practice.  Pratt  &  Whitney  was  chosen  because  of 
the  closeness  in  function  to  an  Air  Force  SPO  and  because  it 
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has  a  quality  initiative,  complete  with  departmental 
measures  of  performance,  established  within  its  structure. 

3.3.1  Interviews  as  the  Prime  Source  of  Information.  It 

is  important  to  choose  data  collection  methods  that  will 

correspond  in  the  tightest  possible  fit  to  the  information 

needed,  maximize  reliability,  and  reduce  error  as  much  as 

humanly  possible  Feldman,  1981:22).  The  choices  of 

research  technl  ,aes  were  not  extensive. 

Case  studies  may  be  based  on  six  different 
sources  of  evidence:  documentation,  archival 
records,  interviews,  direct  observation, 
participant-observation,  and  physical 
artifacts.  (Yin,  1984:78) 

Documentation  and  archival  records  are  good  sources  of 
information  for  determining  what  the  measurements  of  the 
various  white-collar  departments  are,  and  they  were  used  to 
the  extent  that  they  provided  benefit.  But  in  trying  to 
discover  how  the  measures  were  developed,  most  of  the 
written  material  was  of  limited  use. 

Leaving  the  discussion  of  interviews  until  the  end  of 
this  section,  the  next  sources  of  evidence  to  be  evaluated 
is  direct  observation  and  participant-observation.  In 
certain  instances,  direct  observation  provided  a  good 
insight  into  how  well  the  employees  accepted  the  measures 
and  how  often  the  measures  were  used.  But  due  to  the 
limited  on-site  research  time  available,  the  opportunity  did 
not  present  itself  to  directly  observe  any  department 
conduct  a  measurement  development  exercise.  Participant- 
observation,  which  is  a  mode  of  observation  in  which  the 
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investigator  is  more  involved  than  simply  a  passive  observer 
(Yin,  1984:86),  requires  the  same  time  commitment  that 
direct  observation  does.  Also,  for  each  type  of 
observation,  a  sound  theoretical  base  must  be  formed  prior 
to  beginning  the  observation.  For  a  study  of  the 
development  of  departmental  measures  of  performance,  this 
thesis  could  serve  as  the  theoretical  base,  in  which  case, 
the  next  step  would  be  to  prove  that  this  method  works  in 
practice.  However,  due  to  the  time  frame  of  this  thesis,  it 
was  not  possible  to  develop  and  test  the  measurement 
development  recipe  (see  recommendations  for  continuing 
study.  Chapter  6). 

The  use  of  physical  artifacts  for  this  study  is  not 
applicable.  This  involves  the  use  of  a  technological 
device,  a  tool  or  instrument,  a  work  of  art,  or  some  other 
physical  evidence  for  data  gathering  (Yin,  1984:88).  These 
items  were  not  available  during  the  data  collection  phase. 

"One  of  the  most  important  sources  of  case  study 

information  is  the  interview"  (Yin,  1984:82).  Because  the 

information  needed  varied  in  format  from  respondent  to 

respondent  and  was  used  to  validate  literature  information, 

the  focused  interview  was  the  best  way  to  capture  the 

information  and  served  as  the  primary  source  of  evidence. 

The  focused  interview  is  an  interview  in  which 

a  respondent  is  interviewed  for  a  short 
period  of  time  -  an  hour,  for  example.  In 
such  cases,  the  interviews  may  still  remain 
open-ended  and  assume  a  conversational 
manner,  but  the  interviewer  is  more  likely  to 
be  following  a  certain  set  of  questions 
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derived  from  the  case  study  protocol . 

...a  major  purpose  of  such  an  interview 
might  be  simply  to  corroborate  certain  facts 
that  the  investigator  already  thinks  has  been 
established  (but  not  to  ask  about  other 
topics  of  a  broader,  open-ended  nature).  In 
this  situation,  the  specific  questions  must 
be  carefully  worded,  so  that  the  investigator 
appears  naive  about  the  topic  and  allows  the 
respondent  to  provide  a  fresh  commentary 
about  it...  (Yin,  1984:83) 

A  protocol  is  a  guide  for  the  interviewer  or  researcher 
(Yin,  1984:66).  The  protocol  used  in  this  thesis  is  the 
combination  of  Appendix  A  and  Appendix  B. 

3.3.2  Conducting  the  Interviews.  The  previous  section 
established  interviewing  as  the  most  appropriate  data 
collection  method  for  this  case  study.  This  section 
discusses  the  method  used  to  construct  the  interviews. 

The  reasons  Pratt  &  Whitney  was  chosen  for  the  case  study 
were  because  of  its  closeness  in  structure  and  function  to 
the  Air  Force  System  Program  Office  (SPO),  that  it  has  a 
quality  program  in  place,  and  the  fact  that  concrete 
measurement  was  on  going  as  a  performance  evaluation  tool. 
The  closeness  in  structure  and  function  was  the  primary 
reason.  With  potential  applicability  within  a  SPO  being  a 
key  issue  involving  the  results  of  the  research,  it  was 
important  to  choose  a  business  that  had  a  structure  similar 
to  that  of  the  SPO,  or  at  least  departments  within  the 
business  that  performed  functions  resembling  those 
accomplished  in  a  SPO.  When  a  letter  (see  Appendix  B)  was 
sent  to  Pratt  &  Whitney  to  request  managers  for  the 
interviews,  the  managers  requested  were  from  the  areas  of 
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engineering,  finance,  program  management ,  configuration, 
data,  quality  and  manufacturing,  and  logistics.  The  duties 
of  these  departments  closely  resemble  the  duties  of  their 
counterparts  in  a  SPO.  The  quality  program  chairman  at 
Pratt  &  Whitney  succeeded  in  scheduling  managers  from  these 
areas  for  the  interviews,  and  these  managers  formed  a  cross- 
section  representation  of  all  managers  within  the  white- 
collar  work  environment.  A  summary  of  the  information 
letter  sent  to  Pratt  and  Whitney  for  assistance  in 
determining  those  managers  to  be  interviewed  is  Appendix  A. 

Once  again,  the  research  was  looking  for  a  noticeable 
trend  in  measurement  development  procedures,  which  would 
then  be  compared  to  the  literature.  The  interview 
questionnaire  was  developed  to  allow  the  interviewed  manager 
to  state  how  he/she  felt  their  department  developed  their 
measures  of  performance.  The  questions  were  focused  around 
the  integrated  approach  but  were  worded  in  a  manner  that  did 
not  force  the  manager  to  give  an  answer  which  would  coincide 
with  the  integrated  approach.  The  questionnaire  is  Appendix 
B.  The  results  of  the  interviews,  as  well  as  the 
comparisons  to  the  literature,  are  in  Chapter  5. 

3.4  Concern  for  Validity 

C.  William  Emory  defines  validity  in  terms  of  whether  a 

measure  does  what  is  claimed  for  it  (Emory,  1985:115).  Yin 

takes  this  concept  to  greater  depth. 

The  case  study  investigator  also  must 
maximize  four  aspects  of  the  quality  of  any 
design:  (1)  construct  validity  (2) 
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internal  validity  (for  explanatory  or  causal 
case  studies  only)  (3)  external  validity 
(4)  reliability.  (Yin,  1984:27) 

An  explanatory  case  study  is  one  in  which  the  researcher 
is  attempting  to  pose  competing  explanations  for  the  same 
set  of  events  (Yin,  1984:16).  For  example,  does  a  person 
run  faster  because  he  has  better  shoes  or  because  the  shoes 
make  him  psychologically  better?  A  causal  case  study 
attempts  to  show  a  cause  and  effect  relationship  between 
events.  For  example,  if  a  person  runs  faster,  is  it  due  to 
the  new  shoes?  The  case  study  for  this  thesis  was 
descriptive  in  nature,  not  explanatory  or  causal,  since  it 
described  the  development  of  departmental  measures  of 
performance.  Therefore,  construct  validity,  external 
validity,  and  reliability  must  be  addressed. 

Construct  validity,  which  is  establishing  the  correct 
operational  measures  for  the  concepts  being  studied,  or 
using  the  right  ruler,  is  strengthened  by  using  multiple 
sources  of  evidence  (Yin,  1984:36).  For  this  thesis, 
construct  validity  was  strengthened  by  conducting  multiple 
interviews  and  using  appropriate  available  documentation. 

External  validity,  which  is  "establishing  the  domain  to 
which  a  study's  findings  can  be  generalized"  (Yin,  1984:36) 
was  also  addressed  by  interviewing  multiple  departments, 
many  of  which  are  quite  similar  in  function  to  those 
departments  within  a  typical  Air  Force  System  Program 
Office.  This  technique  also  serves  to  encompass  a  variety 
of  management  styles  as  well  as  organizational  differences. 
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Finally,  reliability,  which  is  "demonstrating  that  the 
operations  of  a  study  -  such  as  the  data  collection 
procedures  -  can  be  repeated  with  the  same  results"  (Yin, 
1984:36)  was  addressed  by  using  the  same  interview  protocol 
for  each  interview. 

3.5  Conclusion 

As  discussed  previously,  the  methodology  revolves  around 
two  investigative  questions  involving  two  data  collection 
methods:  an  analysis  of  literature  and  a  case  study.  The 

analysis  of  literature  is  intended  to  provide  the  background 
information  necessary  to  develop  an  integrated  measurement 
development  procedure.  This  integrated  approach  then  serves 
as  the  point  of  comparison  for  the  remainder  of  the  study. 

The  case  study  was  designed  to  ground  the  theoretical 
measurement  development  procedure  in  a  selected  business 
environment.  Because  the  study  was  non-manipulative  in 
nature,  it  fit  well  into  the  case  study  framework. 
Interviewing  was  used  primarily  because  it  formed  the  most 
appropriate  data  collection  method  for  this  research  effort. 
The  data  from  the  interviews  was  compared  with  the 
integrated  approach  from  the  literature  to  determine  if 
departments  that  had  measures  of  performance  followed  the 
literature  principles  when  developing  the  measures. 
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IV.  Analysis  of  Literature 


4.1  Introduction 

This  literature  review  summarizes  the  current  information 
on  the  topic  of  developing  performance  measures  within  a 
white-collar  work  environment  and  identifies  an  integrated 
approach  to  measurement  development.  Throughout  the 
research,  only  six  sources  were  found  containing  measurement 
development  procedures  for  a  white-collar  environment.  In 
this  analysis  of  literature,  these  six  measurement 
development  procedures  are  outlined.  These  procedures  act 
as  references  for  the  subsequent  discussion.  Also  in  this 
section,  the  steps  of  an  integrated  measurement  development 
procedure  based  on  the  six  sources  are  introduced.  This 
summarized  procedure  serves  as  the  basis  of  comparison 
between  the  literature  and  an  industry  example,  as  well  as 
the  basis  for  the  overall  recommendations. 

4.2  Measurement  Development  Procedures  from  Literature 

To  facilitate  reference,  to  form  a  clear  basis  for  the 

discussion  of  the  steps  involved  in  forming  a  performance 
measurement  system,  and  to  form  the  basis  for  the  final 
recommendations,  the  following  performance  development 
procedures  were  found  to  be  applicable. 

Procedure  A  is  from  Productivity  and  Quality: 

Measurement  as  a  Basis  for  Improvement,  by  Everett  E.  Adam, 
et . al .  , 
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Procedure  A:  Measurement  Procedure 
Activities 


1.  Decide  to  measure  productivity 

2.  Select  project  coordinators 

3.  Become  familiar  with  system 

4.  Select  participants;  divide  into  groups 

5.  Define  system  and  establish  boundaries 

6.  Determine  unit  operations 

7.  Generate,  select,  and  rank  deviations 

8.  Determine  final  key  quality  deviations 

9.  Generate  key  quality  deviation  measures 

10.  Collect  technology  and  systems  input 
data 

11.  Edit  measures  into  productivity  ratios 

12.  Clarify  edited  measures  and  revise  as 
necessary 

13.  Put  measures  into  questionnaire  format 

14.  Rank  productivity  measures  by  completing 
questionnaire 

15.  Evaluate  questionnaire  results  and 
produce  final  quality-productivity  ratios. 

(Adam  and  others,  1981:41) 

Procedure  B  is  from  Planning  and  Measurement  in  Your 

Organization  of  the  Future,  by  Sink  and  Tuttle, 

Procedure  B:  Five  Steps  of  Management 
Systems  Analysis  (MSA) 

1.  Gaining  a  better  understanding  of  the 
organizational  system. 

2.  Identification  of  ways  to  improve  the 
performance  of  the  organization  being 
analyzed. 

3.  Classify  information  needs/requirements 
in  relation  to  the  type  of  organizational 
system  that  a  management  team  is  attempting 
to  manage. 

4.  Data  requirements  to  provide  information 
in  Step  3. 

5.  The  data  to  information  transformation. 

How  can  we  best  convert  our  data  into  the 
information  that  has  been  required  by  the 
management  team?  (Sink  and  Tuttle,  1989:192- 
200) 

Procedure  C  is  the  procedure  out  of  the  Office  of 
Management  and  Budget,  which  is: 
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Procedure  C 


1.  Identify  all  customers  of  the  program's 
outputs  -  products  and  services  -  and  those 
customers'  requirements  and  expectations. 

2.  Define  the  entire  workprocess  that 
provides  the  product/service. 

3.  Define  the  value-adding  activities  and 
outputs  that  comprise  the  system. 

4.  Develop  quality  measures  or  indicators. 

5.  Assess  quality  measures.  (OMB,  1989:10- 
16) 

Procedure  D  is  from  the  Productivity  Measurement 
Handbook .  by  William  Christopher. 

Procedure  D 


1.  Identify  the  purpose,  or  mission,  of  the 
unit . 

2.  Identify  the  output  produced  when  this 
purpose  is  successfully  achieved. 

3.  Determine  the  most  successful  measures  of 
this  output. 

4.  Identify  and  measure  inputs. 

5.  Develop  appropriate  productivity 
measures . 

6.  Report  measures  for  the  relevant 
reporting  period  and  monitor  trends  over 
time. 

7.  Do  all  of  the  above  through  a  dialogue 
process.  (Christopher,  1985:195) 

Procedure  E  is  from  Measuring  and  Assessing 

Organizations ,  by  Van  Der  Ven  and  Ferry. 

Procedure  E 


Phase  1:  Evaluation  Prerequisites  -  in  this 
phase,  the  evaluators,  or  those  who  will 
develop  the  measures  are  determined.  This 
book  recommends  using  both  inside  people  and 
people  from  outside  the  organization. 

Phase  2:  Goals  Exploration  -  the  evaluators 
conduct  a  series  of  meetings  with  various 
users,  those  that  have  a  stake  in  the 
organizational  assessment,  to  identify  the 
effectiveness  goals  against  which  the 
organization  will  be  evaluated. 
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Phase  3:  Criteria  Development  -  once  the 
goals  are  determined,  quantitative  measures 
are  broken  out  from  them. 

Phase  4:  Evaluation  Design  -  this  is  the 
phase  when  the  measures  are  tested  in  a  pilot 
program. 

Phase  5:  Evaluation  Implementation  -  the 
tested  goals  are  then  evaluated  for  overall 
effectiveness  of  use. 

Phase  6:  Data  Analysis,  Feedback,  and 
Evaluation  -  this  is  the  continual  assessment 
phase  (Van  Der  Ven  and  Ferry,  1980:31-41). 

Procedure  F  is  from  Company  Productivity:  Measurement 

for  Improvement,  by  Irving  H.  Siegel. 

Procedure  F 


1.  The  decision  to  measure. 

2.  The  task  force  and  its  charter. 

3.  Program  information  and  communication. 

4.  Inventory  of  data  resources  and  skills. 

5.  Auxiliaries:  consultants,  liaison 
officers,  trainees. 

6.  Design  of  measurement  system. 

7.  Installation  and  "debugging". 

8.  Instructions  for  operation  and 
recommendation  for  evaluation  (Siegel, 
1980:45-46) . 

The  following  is  a  summary  of  the  above  material  and 
other  literature  reviewed. 


4.3  The  Integrated  Measurement  Development  Procedure. 

The  integrated  measurement  development  procedure  is 
designed  to  assemble  all  of  the  similar  steps  of  the  above 
procedures  under  a  single  heading.  The  integrated 
measurement  development  procedure  that  encompasses  all  the 
aspects  of  the  six  procedures  from  the  experts  is: 

1.  Step  One:  Establish  a  measurement  team. 


38 


2.  Step  Two:  Obtain  a  clear  understanding  of  the 
organization . 

3.  Step  Three:  Focus  on  measurements  in  the  areas  of 
strategic  importance. 


4. 

Step 

Four: 

Classify  information  needs. 

5. 

Step 

Five : 

Assess  the  measures. 

Sections  4.3.1  through  4.3.5  give  explanations  of  the 
meaning  of  each  step.  Those  sections  are  followed  by  charts 
which  illustrate  the  comparison  between  each  of  the  source 
measurement  development  procedures  and  the  integrated 
approach. 

4.3.1  Step  One:  Establish  a  Measurement  Team.  The 

measurement  team,  or  management  team,  must  be  clearly 

defined.  "Keep  in  mind  that  we  define  a  management  team  to 

be  the  entire  group  of  managers,  staff,  and  other  employees 

that  make  up  a  target  system"  (Sink  and  Tuttle,  1989:233). 

This  general  definition  implies  that  all  members  of  the 

target  system  are  included  on  the  measurement  team.  Siegel 

approaches  the  subject  of  a  measurement  team  this  way: 

The  help  of  employees  could  be  solicited  in 
the  selection  of  eligible  areas  for  special 
scrutiny  and  in  the  selection  of  measurement 
criteria.  (Siegel,  1980:20) 

Often  times,  it  is  impractical  to  select  the  entire  staff  of 

the  target  system  for  the  measurement  team.  According  to 

Sink  and  Tuttle,  the  size  and  make-up  of  the  team  must  be 

tailored  to  the  organization. 

We  suggest  that  for  smaller  organizational 
systems  (i.e.,  approximately  6  to  20)  that 
most,  if  not  all,  of  the  members  of  those 
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organizational  systems  be  involved  in  many  of 
the  steps  of  the  process....  (Sink  and 
Tuttle,  1989:233) 

Referring  to  Procedures  A-D  found  at  the  beginning  of 
section  3.4,  establishing  a  measurement  team  for  the  purpose 
of  developing  the  measures  corresponds  to  steps  2  and  4  of 
Procedure  A,  step  7  of  Procedure  D,  Phase  1  of  Procedure  E, 
and  steps  1-3  of  Procedure  F  (see  Tables  4-9).  This  does 
not  mean  that  the  other  authors  consider  this  step 
insignificant.  Sink  and  Tuttle,  authors  of  Procedure  B,  say 
this  on  the  subject: 

An  early  step  in  the  process  is  the  formation 
of  the  teams  of  people  who  will  be  completing 
various  steps  of  the  process.  We  use  the 
word  teams  because  we  believe  that 
measurement  systems  cannot  be  successfully 
designed,  developed  and  implemented  by 
individual  measurement  analysts.  (Sink  and 
Tuttle : 232 ) 

The  Office  of  Management  and  Budget,  who  supplied  Procedure 

C  quoted  Carl  Thor,  President  of  the  American  Quality  and 

Productivity  Center  when  he  suggested  that 

ideally,  to  ensure  fairness  and  commitment, 
three  actors  --  the  supplier  of  inputs,  the 
employees  of  the  organization  providing  the 
service,  and  the  user/customer  of  the  service 
--  should  construct  the  measures.  (OMB, 

1989:9) 

The  use  of  an  internal  measurement  team,  rather  than  an 
outside  expert  or  guidance  from  elsewhere  in  the 
organization  that  is  not  involved  with  this  target  system, 
adds  to  the  degree  of  buy-in  from  the  employees.  Lehrer 
attributes  the  buy-in  success  to  participative  techniques 
because  they  add  a  new  positive  element  to  what  might  have 
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been  a  sterile  and  possibly  objectionable  exercise  (Lehrer, 
1983:41)  . 

Siegel  states,  however,  that  there  is  some  merit  to  using 

outside  assistance  along  with,  and  as  a  member  of,  the 

measurement  team,  but  adds  a  caveat: 

In  developing  its  program,  a  company  with 
limited  in-house  talent  may  do  well  to  call 
on  outside  consultants.  Such  supplementation 
of  resident  technical  resources  could  reduce 
the  number  of  false  moves,  shorten  the  time 
required  for  design,  and  yield  a  more 
authoritative  result.  On  the  other  hand,  a 
system  has  to  be  a  company's  "own",  and 
responsibility  for  it  cannot  be  delegated  to 
others  by  contract.  (Siegel,  1980:47) 

While  the  outside  assistance  will  not  understand  the 

organization  as  fully  as  someone  who  works  in  the 

department,  they  will  add  an  unbiased  perspective  when 

developing  the  measures  (Van  Der  Ven  and  Ferry,  1980:33). 

The  amount  of  assistance  is  a  decision  the  department  must 

make  when  determining  the  make-up  of  the  measurement  team. 

When  forming  a  team  for  the  purpose  of  developing 
measures  of  performance,  the  users  of  the  organization's 
product  or  service  should  be  considered.  An  error  in 
measurement  construction  is  developing  measures  exclusive  of 
the  users.  Sink  and  Tuttle  and  Lehrer  speak  of  this  common 
mistake.  Sink  and  Tuttle  state  that  "...developing 
measurement  and  evaluation  systems  independent  of 
significant  user  or  management  team  involvement  is  widely 
recognized  as  being  ineffective  in  the  longer  term"  (Sink 
and  Tuttle,  1989:232).  Lehrer  expands  this  idea: 
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Given  that  most  solutions  start  with  an 
analysis  of  outputs  and  objectives, 
consideration  is  needed  of  different  methods 
of  developing  appropriate  measures.  T’.=“ 
common  approach  has  always  been  that  an 
"expert"  will  view  the  process  and  make 
recommendations.  Although  this  is  sometimes 
appropriate  in  dif f icult-to-measure 
situations  in  the  factory,  it  is  a  fairly 
dangerous  approach  in  professional  areas 
where  the  professional  workers  feel  (often 
correctly)  they  know  more  about  the  process 
than  the  so-called  expert.  Another  approach 
would  be  to  have  the  expert  interview 
different  members  of  the  organization  and 
solicit  their  ideas  as  to  appropriate 
measures.  This  has  the  advantage  of  getting 
some  employee  input,  and  it  allows  the 
analyst's  previous  experience  to  be  tempered 
and  adjusted  by  the  special  circumstance  of 
the  particular  organization.  (Lehrer, 

1983:39) 

4.3.2  Step  Two:  Obtain  a  Clear  Understanding  of  the 
Organization.  Obtaining  a  clear  understanding  of  the 
organization  means  the  measurement  team  must  gain  a  complete 
understanding  of  the  goals  and  objectives  of  the 
organizational  system  or  target  system  within  which  the 
measurement  will  take  place.  As  illustrated  in  the 
following  tables,  this  step  includes  identifying  the 
products/services  of  the  organization  and  its  departments, 
and  the  customers,  or  the  recipients  of  the 
products/services . 

A  vigorous  analysis  of  any  white-collar 
function's  effectiveness  must  begin  with 
three  fundamental  questions: 

What  are  we  primarily  here  for? 

Who  relies  on  us? 

How  do  we  know  their  needs  have  been  met? 

(Leth  and  others,  1985:7) 
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Step  two  is  necessary  to  dispell  the  vagueness  that  Boyett 
and  Conn  talked  about  in  their  article,  "Developing  White- 
collar  Performance  Measures",  when  they  said:  "Until  the 
vagueness  can  be  eliminated,  measures  cannot  be  developed" 
(Boyett  and  Conn,  1988:210). 

The  first  step  in  Sink  and  Tuttle's  five-step  Management 
Systems  Analysis  (MSA)  found  in  Procedure  B  above  deals  with 
the  idea  of  gaining  a  better  understanding  of  the 
organizational  system.  This  direction  also  corresponds  to 
steps  3,  5  and  6  from  the  Measurement  Procedure  Activities 
found  in  Procedure  A  steps  3-2  from  the  procedure  which 
came  out  of  the  Office  of  Management  and  Budget,  Procedure 
C,  as  well  as  step  1  from  the  Productivity  Measurement 
Handbook ,  Procedure  D  and  steps  4-5  from  Procedure  F  (see 
Tables  4-9).  Although  not  specifically  stated,  obtaining  a 
clear  understanding  of  the  organization  is  an  important  part 
of  any  goal  exploration  and  is  contained  within  Phase  2  of 
Procedure  f! 

There  are  two  general  areas  that  are  involved  with 
organizational  clarification.  The  first  is  product  or 
service  identification.  "In  approaching  measurement  of 
white  collar  productivity,  the  first  question  to  ask  is  what 
is  the  output  of  the  processes  lor  which  an  employee  is 
responsible"  (Lehrer,  1983:39).  When  the  Air  Force  Human 
Resources  Laboratory  (AFHRL)  discusses  departmental 
measurement,  they  speak  of  the  types  or  categories  of 
results  the  Air  Force  expects  the  organization  to  accomplish 
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( AFHRL ,  1986:14).  Recalling  the  experimental  pilot  program 

conducted  by  the  American  Productivity  Center  to  determine 

how  to  best  conduct  white-collar  measurement,  their  results 

led  to  the  belief  that, 

the  measurement  phase  was  intentionally 
positioned  after  services  and  objectives  were 
identified  so  pilots  could  determine  what  was 
important  to  measure  and  how  to  use  the  data 
meaningfully.  (Leth  and  others,  1985:25). 

The  second  general  area  involved  in  departmental 

clarification  is  customer  identification.  The  Office  of 

Management  and  Budget,  in  their  departmental  measurement 

recipe  (OMB,  1989:10-16),  states  that  identification  of 

customers  is  the  number  one  concern  for  any  measurement 

system.  C.  Jackson  Grayson,  author  of  the  article  "...But 

What  Do  I  Do  Monday  Morning,"  offers  directions  to  Chief 

Executives  desiring  improvement  ideas. 

...visit  ten  customers  by  yourself;  no  staff, 
no  telephone.  Ask  them  a  simple  question: 

'How  are  we  doing?  How  can  we  do  better?' 

Create  an  absolutely  fanatical  focus  on 
customers,  starting  with  yourself. 

(Grayson: 239) 

W.  Edwards  Demming  is  very  clear  on  this  point. 

Everyone  here  has  a  customer.  And  if  he 
doesn't  know  who  it  is  and  what  constitutes 
the  needs  of  the  customer,  and  work  in  the 
cycle  of  adjustment  to  customer's  needs  and 
what  he  can  produce,  then  he  does  not 
understand  his  job.  (Walton,  1986:28-29) 

4.3.3  Step  Three:  Focus  on  Measurements  in  the  Areas  of 
Strategic  Importance.  In  other  words,  once  the  measurement 
team  is  completely  familiar  with  the  organization  or 
department,  it  must  chose  the  focus  of  the  measurements. 
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This  is  analogous  to  the  theory  of  the  Pareto  analysis, 

which  emphasizes  separating  the  important  few  from  the 

trivial  many  (Schonberger  and  Knod,  1988:593).  Measurement 

of  an  entire  organization  is  generally  difficult,  so  key 

areas  must  be  determined. 

It  is  possible  to  rank  the  seven  performance 
criteria  in  terms  of  their  relative 
importance  for  the  target  system.  In 
general ,  this  can  be  done  by  starting  with 
the  performance  criteria  that  are  most 
closely  linked  to  the  strategic  performance 
dimensions.  (Sink  and  Tuttle,  1989:205) 

The  strategic  performance  dimensions  are  those  goals  the 

department  wants  to  concentrate  on  or  those  activities  that 

are  essential  to  the  department. 

Reiterating  an  important  point  from  the  Organization 

Assessment  Instrument,  Table  8, 

Therefore,  in  Phase  2,  the  evaluators  conduct 
a  series  of  meetings  with  various  users  to 
identify  the  effectiveness  goals  they  have 
for  the  organizational  components  being 
assessed.  Users  ... [are] .. .people  within  and 
outside  the  organization  who  have  a  stake  in 
the  organization  assessment  (Van  Der  Ven  and 
Ferry,  1980 : 34) . 

The  customer  should  be  involved  in  the  task  of  focusing  the 
measures  on  the  areas  of  key  importance,  to  the  degree 
possible . 

Determining  the  focus  of  measurements  is  reflective  of 
steps  7-8  from  Procedure  A,  steps  2  and  3  from  Procedure  B, 
step  3  from  Procedure  C,  step  2  from  Procedure  D  and  Phase  2 
from  Procedure  E  (see  Tables  4-9).  Siegel,  author  of 
Procedure  F  concurs  with  this  step  by  saying,  "[The  task 
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force]  will .. .decide  that  some  subgoals  should  be  deferred 
and  others  given  higher  priority"  (Siegel,  1980:51). 

4.3.4  Step  Four:  Classify  Inf ormation  Needs.  Classify 
information  needs  is  the  step  in  which  the  actual  measures 
of  performance  are  determined.  The  measurement  team 
understands  the  department,  and  has  focused  on  the  most 
important  areas  of  that  department,  it  must  now  find 
measures  for  the  key  areas.  Group  techniques  such  as 
brainstorming  and  Nominal  Group  Technique  can  be  used  to 
complete  this  step.  Examples  of  working  measures  in  a 
white-collar  environment  are  contained  in  Chapter  5. 

From  Sink  and  Tuttle's  Management  Systems  Analysis  (MSA), 
Procedure  B,  step  4,  the  classification  of  information  needs 
is  when  the  management  team  collects  the  data.  The  Office 
of  Management  and  Budget,  in  their  procedure  for  measurement 
development.  Procedure  C,  calls  this  step  4,  developing  the 
quality  measures  or  indicators.  Classify  information  needs 
also  summarizes  steps  9-11  of  the  Measurement  Procedure 
Activities  found  in  Procedure  A,  steps  3-5  from  Procedure  D, 
Phase  3  from  Procedure  E,  and  steps  6-7  from  Procedure  F 
(see  Tables  4-9).  Now  that  the  areas  of  focus  have  been 
identified,  the  actual  measures  of  performance  can  be  taken. 

4.3.5  Step  Five:  Assess  the  Measures.  Both  the 
information  gathered  and  the  measures  themselves  must  be 
evaluated.  In  Sink  and  Tuttle's  MSA,  Procedure  B,  assess 
the  measures  is  the  data  transformation  stage,  where  the 
management  team  makes  the  best  use  of  the  information 
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gathered.  That  is  an  assessment  of  the  information. 

One  level  beyond  the  information  assessment  is  the 
evaluation  of  the  measures  themselves.  This  is  the  final 
step  in  the  Office  of  Management  and  Budget  recipe. 

Procedure  C,  step  12  of  the  Measurement  Procedure  Activities 
from  Procedure  A,  step  6  from  Procedure  D,  Phases  4-6  from 
Procedure  E,  and  step  8  from  Procedure  F  (see  Tables  4-9). 
The  Office  of  Management  and  Budget  states  that  the 
measurers  must: 

Evaluate  to  the  degree  which  they  are: 

practical  to  implement 

easy  to  understand 

able  to  drive  desired  behavior 

developed  with  inputs  from  and  consensus 
with  work  groups 

specific  (OMB,  1989:24) 

4.3.6  Tables.  The  following  tables  illustrate  the 
integration  of  the  six  procedures  from  the  literature  into  a 
single  measurement  development  approach. 
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TABLE  4 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  MEASUREMENT  PROCEDURE  ACTIVITIES  (Procedure  A) 


(Adam  and  others,  1981:41) 


INTEGRATED  APPROACH 

MEASUREMENT 

PROCEDURE  ACTIVITIES 

1.  Establish  a  measurement 
team. 

1.  Decide  to  measure 
productivity . 

2.  Select  project 
coordinators . 

4.  Select  participants; 
divide  into  groups. 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

3.  Become  familiar  with 
system. 

5.  Define  system  and 
establish  boundaries. 

6.  Determine  unit 
operations . 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

7.  Generate,  select,  and 
rank  deviations. 

8.  Determine  final  key 
quality  deviations. 

4.  Classify  information 
needs . 

9.  Generate  key  deviation 
measures . 

10.  Collect  technology  and 
systems  data. 

11.  Edit  measures  into 
productivity  ratios. 

5.  Assess  the  measures. 

12.  Clarify  edited  measures 
and  revise  as  necessary. 
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TABLE  5 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  FIVE  STEPS  OF  MANAGEMENT  SYSTEMS 
ANALYSIS  (MSA)  (Procedure  B) 

(Sink  and  Tuttle,  1989:192-200) 


INTEGRATED  APPROACH 

MSA 

1.  Establish  a  measurement 
team. 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

1.  Gaining  a  better 
understanding  of  the 
organizational  system. 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

2.  Identification  of  ways 
to  improve  the  performance 
of  the  organization  being 
analyzed. 

3.  Classify  information 
needs/ requirements  in 
relation  to  the  type  of 
organizational  system  that 
a  management  team  is 
attempting  to  manage. 

4.  Classify  information 
needs . 

4.  Data  requirements  to 
provide  information  in 
step  3. 

5.  Assess  the  measures. 

5.  The  data  to  information 
transformation . 
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TABLE  6 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  PROCEDURE  FROM  THE  OFFICE  OF  MANAGEMENT 
AND  BUDGET  (OMB)  (Procedure  C) 

(OMB,  1989:10-16) 


INTEGRATED  APPROACH  OMB 

1.  Establish  a  measurement 
team. 


2.  Obtain  a  clear  1.  Identify  all  customers 

understanding  of  the  of  the  program’s  outputs 

organization.  and  those  customer's 

requirements  and  expecta¬ 
tions  . 

2.  Define  the  entire 
workprocess  that  provides 
the  product/service. 


3.  Focus  on  measurements  in  3.  Define  the  value- 

the  areas  of  key  importance.  adding  activities  and 

outputs  that  comprise  the 
system. 


4. 

Classify  information 

4. 

Develop  quality  measures 

needs . 

or 

indicators . 

5. 

Assess  the  measures. 

5. 

Assess  quality  measures. 
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TABLE  7 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  PROCEDURE  FROM  THE  PRODUCTIVITY  MEASUREMENT 
HANDBOOK  ( PMH )  (Procedure  D) 


(Christopher,  1985:195) 


INTEGRATED  APPROACH 

PMH 

1.  Establish  a  measurement 
team. 

7.  Do  all  of  the  above 
through  a  dialogue  process 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

1.  Identify  the  purpose, 
or  mission,  of  the  unit. 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

2.  Identify  the  output 
produced  when  this  purpose 
is  successfully  achieved. 

4.  Classify  information 
needs . 

3.  Determine  the  most 
useful  measures  of  this 
output . 

4.  Identify  and  measure 
inputs . 

5.  Develop  appropriate 
productivity  measures. 

5.  Assess  the  measures. 

6.  Report  measures  for  the 
relevant  reporting  period 
and  monitor  trends  over 
time . 
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TABLE  8 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  ORGANIZATIONAL  ASSESSMENT  INSTRUMENT 
(OAI)  (Procedure  E) 


(Van  Der  Ven,  and  Ferry,  1980:31-41) 


INTEGRATED  APPROACH 

OAI 

1.  Establish  a  measurement 
team. 

Phase  1:  Evaluation 
Prerequisites . 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

3 .  Focus  on  measurements  in 
the  areas  of  key  importance. 

Phase  2:  Goal  Exploration 

4.  Classify  information 
needs . 

Phase  3:  Criteria 
Development . 

5.  Assess  the  measures. 

Phase  4:  Evaluation 
Design. 

Phase  5:  Evaluation 

Implementation . 

Phase  6:  Data  Analysis, 
Feedback,  and  Evaluation. 
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TABLE  9 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  COMPANY  PRODUCTIVITY  MEASUREMENT  PROGRAM 
(CPMP)  (Procedure  F) 


(Siegel  1980:45-46) 


INTEGRATED  APPROACH 

CPMP 

1.  Establish  a  measurement 
team. 

1.  The  decision  to 
measure . 

2.  The  task  force  and  its 
charter . 

3.  Program  information  and 
communication. 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

4.  Inventory  of  data 
resources  and  skills. 

5.  Auxiliaries: 
consultants,  liaison 
officers,  trainees. 

3 .  Focus  on  measurements  in 
the  areas  of  key  importance. 

4.  Classify  information 
needs . 

6.  Design  of  measurement 
system. 

7.  Installation  and 
"debugging" . 

5.  Assess  the  measures. 

8.  Instructions  for 
operation  and 
recommendation  for 
evaluation . 

Implementation . 
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4.4  Conclusion 


Many  companies  have  slight  variations  to  this  process, 
but  the  overall  themes  are  consistent.  Rolls-Royce,  for 
example,  uses  Departmental  Task  Analysis  (DTA) ,  in  which 
step  one  is  to  determine  assignment  and  responsibilities. 
Step  two  is  identifying  your  customer,  defined  by  Rolls- 
Royce  and  anyone,  inside  or  outside  the  company,  who 
receives  an  output  from  you.  They  believe  that  customers 
are  the  only  people  who  can  really  evaluate  your 
performance.  Step  three  is  to  determine  the  customer's 
requirements.  Step  four  is  to  determine  activities 
necessary  to  meet  customer  requirements.  This  is  when  they 
ask  if  they  are  choosing  performance  indicators  which  are 
the  easiest  to  measure  or  the  ones  which  are  the  most 
important  to  their  customers  (Wedge,  1989:1-10). 

Ideally,  the  integrated  approach  presented  above  is 
flexible  enough  to  be  adapted  to  any  organization.  Chapter 
5  of  this  thesis  compares  the  integrated  approach  to  how 
members  of  one  firm,  chosen  for  its  similarity  to  an  Air 
Force  SPO,  developed  their  departmental  measures  of 
performance . 
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V.  Results  of  the  Interviews 

5.1  Explanation  of  Results  Format 

This  chapter  analyzes  the  research  data  generated  through 
interviews  with  Pratt  &  Whitney  managers.  Section  5.1 
contains  a  review  of  the  objective  of  the  chapter,  the 
rational  for  the  method  of  evaluation  of  the  data,  a 
presentation  of  the  success  criteria,  and  an  explanation  of 
the  format  of  the  data  summary.  Section  5.2  is  the  data 
summary,  and  section  5.3  discusses  any  additional  findings 
pertinent  to  the  research  but  too  general  for  inclusion  into 
a  particular  interview  summary. 

As  men-,  ioned  in  the  methodology  (Chapter  3),  the  results 
were  tabulated  in  the  form  of  a  comparison  between  the  steps 
found  in  the  integrated  measurement  development  procedure 
discussed  in  Chapter  4  and  the  steps  used  for  measurement 
development  by  the  various  Pratt  &  Whitney  departments.  The 
goal  of  this  comparison  is  to  provide  grounding  in  private 
industry  for  the  information  derived  from  current 
literature,  thereby  adding  validity  to  the  literature 
results . 

The  information  gathered  in  the  Results  of  the  Interviews 
chapter  is  the  manager's  perspective  of  how  the  performance 
measurements  were  developed.  Any  use  of  the  information 
must  be  viewed  in  the  context  in  which  it  was  collected. 

The  questionnaire  (Appendix  B)  was  designed  to  remove  as 
much  of  the  manager's  personal  interpretation  as  possible. 


55 


but  without  the  ability  to  interview  the  entire  department, 
it  is  necessary  to  accept  the  interview  answers  at  face 
value . 

Along  with  providing  insight  into  the  development  of  the 
measures  of  performance,  the  interview  results  provide 
valuable  examples  of  measures  of  performance  used  in 
practice.  These  examples  could  prove  useful  for  any 
department  wishing  to  develop  its  own  measures  of 
performance . 

5.1.1  Rationale  for  the  Method  of  Evaluation.  The  data 
collected  through  the  interviews  were  the  result  of  a  guided 
questionnaire  (Appendix  B)  designed  to  be  flexible  enough  to 
adapt  to  each  respondent  and  their  department’s  unique 
measurement  development  system.  As  a  result,  the 
information  does  not  lend  itself  to  statistical  analysis. 

The  evaluation  procedure,  then,  was  to  look  for  common  ideas 
discussed  throughout  each  interview  and  to  determine  how 
well  the  comments  fit  into  a  format  similar  to  the 
integrated  measurement  development  procedure  from  Chapter  4. 
This  final  transformation  facilitates  the  comparison 
necessary  to  support  or  reject  the  proposition  that  a 
measurement  development  procedure  derived  from  current 
literature  is  applicable,  in  practice,  to  Pratt  &  Witney. 

5.1.2  Success  Criteria.  The  success  criteria  for  the 
interviews  was  100%  compliance  with  the  above-stated 
proposition.  Each  interview  was  evaluated  based  on  this 
success  criteria  when  the  information  gathered  was  compared 
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step  by  step  with  the  integrated  measurement  development 
procedure.  All  discrepancies  are  discussed  in  the 
Recommendations  and  Conclusions  chapter  (Chapter  6). 

5.1.3  Explanation  of  the  Format  for  the  Data  Summary. 
Section  5.2,  the  Data  Summary,  is  laid  out  by  interview.  A 
discussion  of  each  interview  is  included  to  provide  insight 
into  the  type  of  department  involved.  Following  the 
discussion  is  a  table  which  compares  the  measurement 
development  technique  used  by  that  department  to  the 
integrated  approach.  If  text  is  included  in  the  table  next 
to  the  integrated  approach,  it  is  an  explanation  of  that 
department's  measurement  development  step  which  was 
comparable  to  a  particular  step  of  the  integrated 
measurement  technique.  If  no  text  is  presented,  it  means 
that  the  department  did  not  perform  that  step. 

5.2  Data  Summary 

Pratt  &  Whitney  was  chosen  as  a  source  for  the  interviews 
because  its  Government  Engine  Business  has  departments 
similar  in  function  to  those  of  an  Air  Force  System  Program 
Office  (SPO),  and  because,  as  a  result  of  Pratt  &  Whitney's 
quality  initiative,  many  of  the  departments  have  measures  in 
place.  The  following  data  comes  from  twelve  departments 
that  perform  similar  functions  to  their  Air  Force 
counterparts  in  a  SPO  and  have  measures  of  performance  in 
place . 
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5.2.1  Manager  1:  Supervisor  of  Production  Pricing.  The 


Production  Pricing  department  is  responsible  for  preparing 
cost  proposals  for  United  States  military  aircraft  parts. 

The  department  supports  audits  and  fact  findings,  as  well  as 
contract  negotiations.  The  six  members  of  the  department 
produce  both  products  in  terms  of  cost  proposals,  financial 
analyses,  and  budgetary  and  planning  estimates;  and 
services,  in  terms  of  their  support  of  audits,  fact 
findings,  and  negotiations.  The  department's  external 
customers  include  the  United  States  government  and  foreign 
governments.  Its  internal  customer  is  executive  management. 

The  Production  Pricing  department  h-s  three  forms  of 
measurement.  On  the  individual  level,  it  employs  a 
performance  measurement  system  called  the  Performance 
Management  Recognition  and  Rewards  ( PMRR )  method  in  which 
the  manager  negotiates  key  job  requirements,  defined  and 
measurable,  with  the  members  of  the  department.  While  this 
method  is  primarily  designed  for  individual  performance 
measurement,  it  is  also  a  measure  of  department  performance. 
This  will  be  the  only  reference  to  the  use  of  the  PMRR,  but 
it  is  used  throughout  the  company,  and  each  department 
interviewed  participates  in  this  system.  For  the  rest  of 
the  discussions,  it  will  be  understood  that  the  PMRR  is  a 
measurement  system  in  addition  to  those  presented. 

Another  measurement  of  this  department  is  a  proposal 
tracking  system  for  spare  parts  and  support  equipment.  This 
tracking  system  assesses  how  well  the  department  is  doing 
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processing  proposals  against  negotiated  due  dates.  The 
information  from  the  tracking  system  is  used  by  the  Air 
Force  Plant  Representative  Office  (AFPRO)  to  determine  the 
performance  of  the  contractor. 

The  final  measurement  system  used  by  the  Pricing 
department  is  a  government  audit  of  the  department's 
estimating  procedures.  The  criteria  used  in  this  measure 
are  dictated  by  the  government  and,  while  the  department  is 
aware  of  the  criteria,  it  did  not  participate  in  developing 
the  measures. 
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TABLE  10 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  THREE  PRODUCTION  PRICING  APPROACHES 


INTEGRATED  APPROACH  PRODUCTION  PRICING 


1.  Establish  a  measurement  A  measurement  team  was  not 

team.  established,  but  each 

measure  was  designed  in 
conjunction  with  the 
affected  members  of  the 
department . 


2.  Obtain  a  clear 
understanding  of  the 
organization. 


3.  Focus  on  measurements  in 
the  areas  of  key  importance. 


4.  Classify  information 
needs . 


5.  Assess  the  measures. 


The  members  of  the 
department  are  clear  on 
what  is  required  of  them 
as  well  as  the  purpose  of 
the  department.  This  is 
particularly  reinforced 
in  the  PMRR  measurement 
system. 

Each  measurement  system 
focuses  on  the  needs  of 
customers,  which  the 
department  feels  is  the 
area  of  key  importance. 

The  measures  of  each  system 
reflect  the  performance  in 
each  of  the  key  areas 
defined  in  step  three. 

The  manager  continually 
assesses  the  measures  and 
works  to  improve  any 
deficiencies . 


5.2.2  Manager  2:  Supervisor  of  the  Travel  Audit  Group. 
The  six  members  of  the  Travel  Audit  Group  are  responsible 
for  reviewing  the  expense  reports  for  compliance  with 
company  regulations,  as  well  as  military  and  Internal 
Revenue  Service  requirements.  The  department  handles  30,000 
to  32,000  reports  per  year.  The  product  of  the  department 
is  the  information  provided  to  the  finance  department  for 
their  records  and  the  processing  of  the  reports  so  that  the 
travellers  can  be  reimbursed.  Its  customers  are  primarily 
internal . 

The  measurement  system  for  the  Travel  Audit  Group  is 
relatively  simple.  It  uses  a  three  box  system.  All  expense 
reports  that  arrive  at  the  department  are  placed  in  the 
first  box  on  the  first  day  of  their  arrival.  All  expense 
reports  not  processed  on  the  first  day  move  to  the  second 
box  on  the  second  day.  All  expense  reports  not  processed  on 
the  second  day  move  to  the  third  box  on  the  third  day. 

Since  the  department  set  a  four  day  commitment  to  process 
the  reports,  no  one  in  the  department  goes  home  until  the 
third  box  is  empty. 
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TABLE  11 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  TRAVEL  AUDIT  GROUP  APPROACH 


INTEGRATED  APPROACH  TRAVEL  AUDIT  GROUP  APPROACH 

1.  Establish  a  measurement  The  manager  did  not  form  a 

team.  team  to  determine  this 

measurement  system.  The 
idea  was  his,  and  the 
members  of  the  department 
agreed  to  go  along  with  it, 
so  it  was  not  necessary  to 
set  up  a  team  to  come  up 
with  a  different  idea. 


2.  Obtain  a  clear  The  mission  of  the 

understanding  of  the  department  is  clear  to  all 

organization.  who  participate  in  the 

measurement  system,  who  are 
the  same  people  who 
approved  the  system  - 
or  at  least  acquiesced. 


3.  Focus  on  measurements  in  The  important  area  for  the 

the  areas  of  key  importance.  department  is  return  time 

for  the  expense  reports. 


4.  Classify  information  The  system  provides  the 

needs.  required  information  to 

determine  the  department's 
performance  in  the  key 
areas . 


o.  Assess  the  measures. 


The  manager  assesses  the 
system  continuously. 
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5.2.3  Manager  3;  Manager  of  a  Systems  Engineering 


Group .  The  Systems  Engineering  Group  is  responsible  for  the 
reliability,  maintainability,  safety,  configuration,  and 
data  management  aspects  of  one  of  the  company's  primary 
products.  The  twelve  members  of  the  group  also  perform 
budget  and  schedule  tracking  for  their  customers,  the 
government  and  the  company's  senior  management.  The  product 
of  the  Systems  Engineering  Group  is  primarily  the  support 
services  they  provide. 

The  group  performs  several  tasks  and  it  is  only  logical 
that  they  have  several  ongoing  measurement  systems  for  the 
variety  of  tasks  accomplished.  The  group  is  measured  for 
delivery  of  contract  data  and  contract  required  reports 
against  the  negotiated  commitments  with  the  government.  It 
internally  tracks  budget  versus  actuals  in  the  financial 
arena.  The  manager  also  keeps  tabs  on  the  responsiveness  of 
the  department  members  to  the  customer,  although  the 
department  does  not  have  a  formal  measurement  system  for 
■  his . 
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TABLE  12 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  SYSTEMS  ENGINEERING  GROUP  APPROACH 


INTEGRATED  APPROACH 

SYSTEMS  ENGINEERING  APPROACH 

1.  Establish  a  measurement 
team. 

The  Systems  Engineering 
Group  did  not  formally 
commission  a  measurement 
team.  However,  all 
the  measurements  it  uses 
are  negotiated  with  the 
participating  employees 
to  achieve  commitment. 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

Support  requirements,  the 
purpose  of  the  department, 
are  worked  out  with  the 
employees.  They  are  clear 
on  who  their  customer  is. 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

The  areas  of  key 
importance  are  the  centers 
for  each  measurement 
system. 

4.  Classify  information 
needs . 

Each  measurement  system 
that  the  group  uses 
provides  the  necessary 
information  for  judging 
performance  in  the  key 
areas . 

5.  Assess  the  measures. 

The  manager  feels  that 
the  measures  work  well  and 
would  take  steps  to  change 
them  if  they  did  not. 
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5.2.4  Manager  4:  Manager  of  the  Hot  Section  Engineering 


Department .  The  Hot  Section  Engineering  Department  is 
responsible  for  the  design,  fabrication,  and  development  of 
the  high  performing  components  for  one  of  the  primary 
engines  of  the  company.  The  ten  members  of  the  department 
have  external  customers  in  the  government  and  internal 
customers  in  the  engine  program  office. 

The  Hot  Section  Engineering  Department  uses  a  measurement 
system  called  the  Systems  Engineering  Detailed  Schedules 
(SEDS).  SEDS  tracks  the  department's  performance  against 
established  milestones  that  are  on  contract.  The  department 
also  tracks  how  well  it  aligns  with  its  actual  expenditure 
versus  the  budget  expenditures,  and  this  information  is 
reviewed  with  the  participating  engineers. 
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TABLE  13 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  HOT  SECTION  ENGINEERING  DEPARTMENT  APPROACH 


INTEGRATED  APPROACH 

ENGINEERING  APPROACH 

1.  Establish  a  measurement 
team. 

The  department  did  not 
commission  a  formal 
measurement  team.  But 
original  input  from  the 
department  members  was 
required  for  the  SEDS 
measurement  system  and 
all  requirements  are 
reviewed  with  the 
participants . 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

The  members  of  the 
department  are  clear  on 
what  is  required  of  them 
and  what  the  purpose  of 
department  is.  This 
information  is  reviewed 
twice  a  year.  This 
activity  is  important 
because  these  are  the 
people  who  are  involved 
in  setting  the  measures. 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

SEDS  is  in  alignment  with 
what  the  program  deems  is 
important . 

4.  Classify  information 
needs . 

The  SEDS  system  and  the 
budget  versus  actuals 
reviews  provide  the 
necessary  information  to 
determine  the  department’s 
performance . 

5.  Assess  the  measures. 

The  measures  are 
continuously  reviewed  for 
aptness  and  would  be 
changed  if  they  were  not 
appropriate . 

66 


5.2.5  Manager  5:  Supervisor  of  the  Engineering  Turbine 


Durability  Department.  This  department  is  responsible  for 
the  thermal,  fluid  flow  and  life  analyses  of  all  military 
engines  in  service  today.  There  are  twelve  members  of  the 
department  whose  products  include  airfoil  technical  drawings 
as  well  as  support  service  for  advice  on  turbine  durability 
related  items,  such  as  manufacturing  and  field  problems. 

The  customers  of  the  department  are  project  engineering, 
manufacturing,  product  support,  and  government  services  (Air 
Force,  Navy,  and  Army). 

The  measurement  systems  in  place  are  extremely  customer 
oriented.  Improved  design  is  indicated  by  factors  like 
improved  man-hour  repair  rates  and  customer  satisfaction. 
Both,  in  turn,  are  indicators  of  performance,  although  there 
is  no  formal  system  in  place  to  measure  them.  Whenever 
there  is  a  field  problem,  however,  milestones  are  negotiated 
and  the  department  is  measured  against  those  milestones. 
Finally,  this  department  also  tracks  budgeted  expenditures 
against  actuals. 
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TABLE  14 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  DURABILITY  DEPARTMENT  APPROACH 


INTEGRATED  APPROACH 

1.  Establish  a  measurement 
team. 


DURABILITY  DEPARTMENT  APPROACH 

A  formal  measurement  team 
was  not  established.  The 
requirements  of  the 
department,  however,  are 
negotiated  with  the 
participating  members. 


2.  Obtain  a  clear 
understanding  of  the 
organization. 


Each  member  of  the  depart¬ 
ment  is  clear  about  the 
requirements  of  their  job 
and  what  is  expected  of 
them  personally. 


3.  Focus  on  measurements  in  For  this  department,  the 

the  areas  of  key  importance.  key  areas  are  customer 

satisfaction,  fixing  field 
problems,  and  budget 
constraints.  These  are  the 
areas  against  which  the 
members  are  evaluated. 


4.  Classify  information 
needs . 


Each  of  the  measurement 
systems  in  place  provide 
the  information  necessary 
to  determine  the  level 
of  performance  for  the 
department  in  those  key 
areas . 


5.  Assess  the  measures.  Because  the  measures  are 

unique  in  many  cases  to 
the  problem  at  hand, 
continual  assessment  of 
measures  is  not  necessary. 
For  those  measures  that 
are  constant,  they  are 
monitored  for  adequacy. 
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5.2.6  Manager  6:  Supervisor  of  the  Enhanced  Flow 


Compressor  Test  Group.  This  department  is  responsible  for 
fan  and  compressor  test  contracts.  The  sixteen  members  of 
the  department  run  government  contracted  tests,  as  well  as 
in  house  tests.  The  customers  include  divisions  within 
Pratt  &  Whitney  and  the  government. 

The  performance  measures  for  this  department  are  test 
schedules,  budget  versus  actuals,  and  customer  satisfaction. 
Formal  measures  for  customer  satisfaction  are  not  in  place, 
but  the  other  two  measures  are  negotiated  with  the 
department  as  part  of  the  test  contracts. 
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TABLE  15 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  ENHANCED  FLOW  COMPRESSOR  GROUP  APPROACH 


INTEGRATED  APPROACH 

COMPRESSOR  GROUP  APPROACH 

1.  Establish  a  measurement 
team. 

A  formal  measurement  team 
does  not  exist.  However, 
the  members  of  the 
department  are  participants 
in  the  development  of  the 
contractual  statements  of 
work . 

2.  Obtain  a  clear 
understanding  of  the 
organization . 

Through  the  writing  of 
statements  of  work,  the 
members  of  the  department 
get  a  clear  understanding 
of  what  is  required  of 
them. 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

Once  again,  the  statements 
of  work  play  a  key  factor. 
Since  the  work  from  this 
department  is  laid  out, 
along  with  the  measures,  in 
these  documents,  they 
automatically  focus  the 
measures  where  the 
customer  chooses. 

4.  Classify  information 
needs . 

The  information  produced 
by  the  measures  is  what  is 
required  for  an  assessment 
of  the  department. 

5.  Assess  the  measures. 

Since  the  measures  are 
not  the  same  for  each 
contract,  assessment  is 
ongoing  with  each  statement 
of  work  creation. 
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5.2.7  Manager  7:  Supervisor  of  the  Computational 


Structural  Methods  Group  (CSMG).  The  mission  cf  this  group 
is  to  produce  advanced  structural  analysis,  design  tool  and 
data  management  methods  that  will  satisfy  the  customer’s 
requirements  for  capable,  efficient  and  usable  computer 
programs.  There  .are  ten  people  who  work  for  the  supervisor. 
The  customers  of  this  department  are  internal :  Pratt  & 
Whitney's  Government  Engine  Business,  its  Commercial  Engine 
Business,  and  Pratt  &  Whitney  Canada. 

This  particular  department  has  difficulty  identifying 
appropriate  measures  of  performance.  Whenever  programming 
is  involved,  it  is  easy  to  try  measuring  performance  by  the 
number  of  lines  of  code  written,  or  number  of  programs 
written.  However,  the  manager  did  not  believe  that  these 
are  effective  measures  because  they  did  not  measure  the 
performance  of  the  department,  just  the  quantity  of  output. 
Internal  to  the  Computational  Structural  Methods  Group,  it 
monitors  how  often  each  program  it  writes  is  usi  i.  This 
measure  is  used  to  determine  what  the  customers  requirements 
are  ar i  whether  they  are  being  met.  Pratt  &  Whitney  also 
has  a  Software  Quality  Assurance  group  whose  job  it  is  to 
determine  if  the  programs  written  conform  to  the 
specifications  Pratt  has  adopted.  The  Computational 
Structural  Methods  Group  provides  inputs  into  these 
measures . 
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TABLE  16 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  COMPUTATIONAL  STRUCTURAL  METHODS  GROUP  APPROACH 


INTEGRATED  APPROACH 

1.  Establish  a  measurement 
team. 


CSMG  APPROACH 

CSMG  did  not  establish  a 
formal  measurement  team. 
However,  each  member  of  the 
department  participated 
in  measuring  the  use  rate 
of  their  programs  and  the 
department  on  the  whole 
was  involved  with 
establishing  the  measures. 


2.  Obtain  a  clear  The  CSMG  department  had  a 

understanding  of  the  formal  mission  statement 

organization.  signifying  that  it  had 

a  firm  grasp  on  what  the 
duties  of  the  department 
are  as  well  as  who  its 
customers  are. 


3.  Focus  on  measurements  in  Since  this  is  a  service 

the  areas  of  key  importance.  department,  customer 

satisfaction  is  an  area  of 
key  importance.  Also,  as 
a  functioning  unit  within 
Pratt,  following  Pratt's 
policies  is  another  area 
of  key  importance. 


4.  Classify  information 
needs . 


The  measures  it  employed 
provided  the  necessary 
information  to  assess  the 
performance  of  the  depart 
ment . 


5.  Assess  the  measures.  After  assessing  the 

measures  with  the  manager, 
these  were  the  best 
available.  He  periodically 
assesses  them  for  validity. 
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5.2,8  Manager  8:  Supervisor  of  the  Complex  Support 
Equipment  Group.  This  department  is  responsible  for  the 
engine  test  stands,  the  test  equipment,  and  the  horoscopes 
delivered  to  the  Air  Force.  The  thirty  members  of  the 
department  provide  the  aforementioned  products  as  well  as 
the  service  of  supporting  these  items  once  they  are  in  the 
Air  Force  inventory.  The  customers  include  the  Air  Force 
and  the  program  managers  within  Pratt  $.  Whitney. 

The  measurement  system  for  this  department  involves 
meeting  the  contract  schedule  for  delivering  the  support 
equipment  to  the  Air  Force.  These  contractual  requiremen .3 , 
which  are  negotiated,  are  the  criterion  against  which  both 
the  internal  and  external  customers  measure  the  department. 

This  particular  department  had  a  typical  example  of  how 
it  arrived  at  their  measures  cf  performance.  A  unique  piece 
of  test  equipment  required  a  long  term  solution  to  its 
delivery  problems.  The  department  formed  a  team  to  solve 
the  problems  and  come  up  with  some  measures  of  performance 
to  compliment  the  contractual  requirements.  The  team  worked 
with  the  Air  Force  to  determine  actual  needs  and  worked  to 
meet  those  needs.  The  measures  used  by  this  team  were  for 
the  purpose  of  determining  performance  as  well  as 
improvement.  The  corrective  action  worked.  The  problems 
were  solved  and  the  team  continued  to  use  the  measures  as 
performance  indicators. 
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TABLE  17 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  SUPPORT  EQUIPMENT  (SE)  GROUP  APPROACH 


INTEGRATED  APPROACH 

SE  GROUP  APPROACH 

1.  Establish  a  measurement 
team. 

In  this  case,  a  team  was 
used  to  come  up  with  the 
best  measures  of 
performance  for  its 
sector  of  the  department. 
This  sector  has  its  own 
goals,  consistent  with  the 
goals  of  the  department. 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

This  department  has  a  clear 
mission.  Each  member 
understands  what  their  role 
is  within  the  department. 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

Since  the  number  one 
concern  of  the  department 
is  customer  satisfaction, 
achieved  through  on-time 
deliveries,  the  department 
is  focusing  on  the  areas 
of  key  importance. 

4.  Classify  information 
needs . 

The  department  members  use 
a  war  room  to  status  the 
performance  of  each 
program  within  the  depart¬ 
ment.  The  measures  provide 
the  information  necessary 
to  update  each  status. 

This  information  is  then 
passed  to  management  to 
assess  the  performance  of 
the  department. 

5.  Assess  the  measures. 

The  department  had  its 
choice  of  whatever  measures 
it  wanted.  It  chose  the 
war  room  for  a  summary  of 
the  measures  and  it 
continually  assess  the 
measures . 
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5.2.9  Manager  9:  Supervisor  of  the  System  Integrity 
Group .  This  is  the  internal  systems  audit  group  within 
Pratt  &  Whitney  that  determines  whether  the  departments  are 
following  procedures.  While  the  group  does  not  have  prime 
examples  of  departmental  measures,  the  purpose  of  discussing 
this  department  is  because  it  employs  many  of  the  same 
measurement  development  procedures  as  the  departments  it 
audits . 

The  System  Integrity  Group  tracks  the  quality  measures 
for  Pratt  &  Whitney.  The  measures  are  based  on  goals.  The 
results  of  the  measures  are  used  by  the  AFPRO  to  determine 
the  performance  of  Pratt  &  Whitney  as  a  the  whole.  Each  of 
the  measures  are  negotiated,  so  that  the  departments  being 
evaluated  have  a  large  voice  in  determining  how  they  will  be 
evaluated.  All  products  and  services  are  identified  and 
each  of  the  quality  indicators,  or  measures,  come  complete 
with  explanations. 

This  discussion  is  an  evaluation  of  a  system  used 
throughout  Pratt  &  Whitney.  It  is  not  an  evaluation  of  a 
particular  department's  measurement-  development  system. 
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TABLE  18 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  PRATT  &  WHITNEY  APPROACH 


INTEGRATED  APPROACH 

PRATT  APPROACH 

1.  Establish  a  measurement 
team. 

Pratt  &  Whitney  does  not 
form  a  team  of  the  entire 
Government  Engine 

Business  when  establishing 
its  measures  of 
performance.  It  formed 
the  System  Integrity  Group 
as  an  internal  audit  group 
to  come  up  with  measures. 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

The  products  and  services 
of  each  department  are 
identified.  This  allows 
the  System  Integrity  group 
to  obtain  a  clear 
understanding  of  the 
department  being  evaluated 

3.  Focus  on  measurements  in 
the  areas  of  key  importance. 

After  the  products  and 
services  are  identified, 
the  measures  are  designed 
to  focus  on  performance 
in  those  key  ares. 

4.  Classify  information 
needs . 

The  measures  are  used  at 
the  Product  Assurance  and 
Contractual  Effectiveness 
meetings.  The  measures, 
and  the  information  gained 
from  the  measures  must 
contribute  to  accurate 
assessments  of  each 
department's  performance. 

5.  Assess  the  measures. 

The  measures  are  assessed 
for  aptness  by  the  group 
supervisor,  AFPRO,  and 
Pratt  &  Whitney. 

76 


5.2.10  Manager  10:  Manager  of  Operations  and  Quality 


Assurance .  This  purely  service  department  performs 
dimensional  inspections,  assembly  test  and  overhaul 
inspections,  and  in-process  inspections  of  engines  in 
production.  The  customers  of  the  110  member  shop  include 
the  assembly  shop,  the  engineers,  and  the  Air  Force. 

There  are  three  measures  of  performance  for  the 
inspectors.  First,  a  re-inspector  reviews  the  work  of  the 
inspector.  Not  all  the  work  is  reviewed.  The  re-inspector 
chooses  pieces  at  random  and  inspects  those  pieces  against 
the  same  list  the  first  inspector  used. 

Second,  the  inspectors  are  timed  to  determine  how  long  it 
takes  them  to  perform  the  inspection  tasks. 

Finally,  there  are  inspections  of  the  engine  when  it 
arrives  at  its  destination.  This  inspection  determines, 
from  the  customer's  perspective,  how  well  the  Pratt  & 
Whitney  inspectors  did  at  finding  any  manufacturing  errors. 

The  goals  of  these  measures  are  self-imposed,  based  on 
customer  needs.  They  are  reviewed  by  higher  management 
periodical  1 y . 
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TABLE  19 


COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  INSPECTION  GROUP  APPROACH 


INTEGRATED  APPROACH  INSPECTION  GROUP  APPROACH 

1.  Establish  a  measurement  The  inspection  group  did 

team.  commission  a  team  to 

determine  the  measurement 
needs  of  the  department. 
Each  measurement  goal  is 
self-imposed  by  the 
department  and  the 
inspectors  know  what  the 
goals  are. 


The  members  of  the 
inspection  department  are 
clear  on  the  mission  of  the 
department . 


3.  Focus  on  measurements  in  The  area  of  key  importance 

the  areas  of  key  importance.  for  this  group  is  to  not 

allow  manufacturing  errors 
to  get  by  undetected  to 
the  customer.  The  measures 
developed  focus  on  this 
area . 


The  measures  are  designed 
to  provide  information  on 
the  department's 
performance  in  the  key 
area . 


5.  Assess  the  measures.  The  measures  are  assessed 

and  they  appear  to  work. 


4.  Classify  information 
needs . 


2.  Obtain  a  clear 
understanding  of  the 
organization . 
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5.2.11  Manager  11:  Program  Manager  of  the  J58  Engine 


Group .  This  department  has  overall  program,  repair, 
overhaul,  and  field  support  responsibility  for  the  J58 
engine.  There  are  forty-eight  members  of  the  group, 
reporting  to  the  program  manager,  that  perform  tasks  in  the 
assembly,  logistics,  operations,  and  field  arenas.  The 
customer  for  this  group  is  the  Air  Force. 

The  primary  measurement  system  for  this  group  is  called 
Management  Tracking  Parameters  (MTP).  The  idea  behind  MTP 
is  to  set  goals  for  critical  items  and  track  the  items  to 
those  goals.  The  employees  are  involved  with  goal  setting. 
These  goals  are  then  reviewed  at  the  Quarterly  Program 
Management  Reviews. 


TABLE  20 

COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 
AND  THE  J58  ENGINE  GROUP  APPROACH 


INTEGRATED  APPROACH  J58  ENGINE  GROUP  APPROACH 

1.  Establish  a  measurement  The  measurement  team  for 

team.  this  program  is  the  entire 

J58  engine  group. 


2.  Obtain  a  clear 
understanding  of  the 
organization. 


3.  Focus  on  measurements  in 
the  areas  of  key  importance. 


The  team  has  a  clear 
understanding  of  the 
program  and  its  goals. 


The  measures  focus  on  the 
critical  items  of  the 
program. 


4.  Classify  information  The  team  generated  measures 

needs.  to  provide  information 

about  the  key  areas. 


5.  Assess  the  measures. 


The  measures  are  assessed. 
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5.2.12  Manager  12:  Chief  of  Engineering  Drafting.  This 


department  is  responsible  for  engineering  hardware  drawings. 
The  five  who  work  for  the  Chief  of  Engineering  Drafting  have 
customers  in  the  design  engineering  departments  and  the 
process  planning  departments.  The  primary  product  are  the 
drawings . 

The  measurement  system  used  by  this  department  is  a  self- 
imposed  inspection  of  the  drawings  for  errors.  The  errors 
in  the  drawings  would  cause  production  of  bad  hardware.  The 
goal  of  the  department  is  a  2%  error  rate. 


TABLE  21 

COMPARISON  BETWEEN  THE  INTEGRATED  APPROACH 

AND  THE  ENGINEERING  DRAFTING  GROUP  APPROACH 

INTEGRATED  APPROACH 

DRAFTING  GROUP  APPROACH 

1.  Establish  a  measurement 

This  department  did  not 

team. 

formally  commission  a 
measurement  team.  However 
the  members  of  the 
department  have  bought  in 
to  the  measures  being  used 

2.  Obtain  a  clear 

The  people  involved  with 

understanding  of  the 

the  measurement  system 

organization . 

have  a  clear  understanding 
of  the  products  and 
customers  of  the 
department . 

3.  Focus  on  measurements  in 

The  customer  is  the  area 

the  areas  of  key  importance. 

of  key  importance  for  this 
department.  The  measures 
are  designed  to  measure 
customer  satisfaction. 

4.  Classify  information 

The  measures  provide  the 

needs . 

necessary  information  to 
assess  performance. 

5.  Assess  the  measures. 

The  measures  are  evaluated 
and  they  are  effective. 
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5.3  Additional  Information 


Many  of  the  departments  interviewed  had  groups  within  the 
departments  that,  although  they  fell  under  the  same  general 
mission,  performed  different  tasks.  It  is  for  this  reason 
that  some  of  the  departments  needed  many  measures  of 
performance . 
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VI.  Conclusions  and  Recommendations 


6.1  Introduction. 

The  final  chapter  summarizes  the  results  of  research, 
presents  overall  findings,  and  provides  recommendations  for 
further  study.  The  summary  of  the  results  will  follow  the 
format  of  data  collection.  It  will  first  entail  a 
discussion  of  the  literature,  followed  by  a  discussion  of 
the  interviews. 

6.2  Summary  of  the  Results  of  the  Research. 

6.2.1  Summary  of  the  Analysis  of  Literature.  The  first 
observation  found  when  compiling  the  information  from  the 
literature  review  was  that  many  of  the  sources  simply 
indicated  what  measure  would  be  best  for  a  white-collar 
environment,  rather  than  expounding  on  the  best  method  for 
developing  the  measures.  An  example  of  this  comes  from  a 
book  called  White  Collar  Waste:  Gain  the  Productivity  Edge, 
by  Val  Olson.  This  book  discusses  specific  measures 
associated  with  planning  the  use  of  time;  for  instance, 
comparing  the  actual  use  of  time  against  the  planned  use  of 
time  (Olson,  1983:153). 

The  second  observation  found  when  researching  the 
literature  was  that  the  recipes  were  not  strictly  targeted 
to  measure  performance.  Procedure  A  discussed  measuring  for 
quality  and  productivity.  Procedure  B  was  more  general, 
measuring  performance.  Procedure  C  referred  to  measuring 
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quality.  Procedure  D  prescribed  measuring  productivity. 
Procedure  E  talks  about  measuring  performance.  Procedure  F 
describes  measuring  productivity.  However,  it  appears  that, 
whether  discussing  performance  or  the  related  aspects  of 
performance,  the  procedures  proposed  to  develop  measures  are 
very  similar. 

The  primary  goal  of  this  thesis  was  to  provide  a 
generalized  measurement  development  procedure.  The 
literature  provided  the  root  source  for  the  integrated 
procedure  that  is  proposed.  Each  of  the  six  sources 
discussed  in  the  analysis  of  literature  went  into  great 
detail  supporting  the  steps  necessary  to  develop  white- 
collar  departmental  measures.  The  following  five  tables 
(Tables  22-26)  are  a  review  of  how  their  suggestions  fit 
into  the  steps  of  the  integrated  approach.  The  title  of 
each  table  is  the  integrated  approach  step;  and  the 
information  contained  in  the  table  is  the  evidence  from  each 
of  the  authors  that  supports  the  integrated  step.  The 
information  that  follows  is  a  tabular  formulation  of  the 
text  contained  in  Chapter  4  and  is  a  useful  review  of  the 
development  of  the  integrated  approach. 

What  the  illustrate  is  that  each  of  the  steps  contained 
in  the  integrated  approach  is  supported  by  each  of  the 
sources.  For  this  reason,  the  integrated  measurement 
development  approach  would  seem  to  have  value  as  a  summary 
method  to  develop  departmental  measures  of  performance  in 
any  white-collar  environment. 
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TABLE  22 


STEP  1:  ESTABLISH  A  MEASUREMENT  TEAM 


Procedure  A  from  Adam,  et .  al . : 

2.  Select  project  coordinators. 

4.  Select  participants;  divide  into  groups. 

Procedure  B  from  Sink  and  Tuttle: 

"An  early  step  in  the  process  is  the  formation  of  the 
teams  of  people  who  will  be  completing  various  steps 
of  the  process"  (Sink  and  Tuttle,  1989:232). 

Procedure  C  from  the  Office  of  Management  and  Budget: 
"Ideally,  to  ensure  fairness  and  commitment,  three 
actors  --  the  supplier  of  inputs,  the  employees  of 
the  organization  providing  the  service,  and  the 
user/customer  of  the  service  --  should  construct 
the  measures"  (OMB,  1989:9). 

Procedure  D  from  Christopher: 

7.  Do  all  of  the  above  through  a  dialogue  process. 
Procedure  E  from  Van  Per  Ven: 

Phase  1:  Evaluation  Prerequisites. 

Procedure  F  from  Siegel: 

1.  The  decision  to  measure. 

2.  The  task  force  ard  its  charter. 

3.  Program  information  ard  communication. 
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TABLE  23 


STEP  2:  OBTAIN  A  CLEAR  UNDERSTANDING  OF  THE  ORGANIZATION 


Procedure  A  from  Adam,  et .  al . : 

3.  Become  familiar  with  system. 

5.  Define  system  and  establish  boundaries. 

6.  Determine  unit  operations. 

Procedure  B  from  Sink  and  Tuttle: 

1.  Gaining  a  better  understanding  'f  the  organizational 
system. 

Procedure  C  from  the  Office  of  Management  and  Budget: 

1.  Identify  all  customers  of  the  program’s  outputs  and 
those  customer's  requirements  and  expectations. 

2.  Define  the  entire  workprocess  that  provides  the 
product/ service. 

Procedure  D  from  Christopher: 

1.  Identify  the  purpose,  or  mission,  of  the  unit. 
Procedure  E  from  Van  Per  Ven: 

This  step  is  part  of  Phase  Two:  Goal  Exploration. 
Procedure  F  from  Siegel : 

4.  Inventory  of  data  resources  and  communication. 

5.  Auxiliaries:  consultants,  liaison  officers, 
trainees . 
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TABLE  24 


STEP  3:  FOCUS  ON  MEASUREMENTS  IN  THE  AREAS  OF  KEY 

IMPORTANCE 


Procedure  A  from  Adam,  et.  al . : 

7.  Generate,  select,  and  rank  deviations. 

8.  Determine  final  key  quality  deviations. 

Procedure  B  from  Sink  and  Tuttle: 

2.  Identification  of  ways  to  improve  the  performance 
of  the  organization  being  analyzed. 

3.  Classify  information  needs/requirements  in  relation 
to  the  type  of  organizational  system  that  a  management 
team  is  attempting  to  manage. 

Procedure  C  from  the  Office  of  Management  and  Budget: 

3.  Define  the  value-adding  activities  and  outputs  that 
comprise  the  system. 

Procedure  D  from  Christopher: 

2.  Identify  the  output  produced  when  this  purpose  is 
successfully  achieved. 

Procedure  E  from  Van  Per  Ven: 

Phase  2:  Goal  exploration. 

Procedure  F  from  Sieael: 

"[the  task  force]  will  decide  that  some  subgoals  should 
be  deferred  and  others  given  higher  priority" 

(Siegel,  1980:51). 
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TABLE  25 


STEP  4:  CLASSIFY  INFORMATION  NEEDS 


Procedure  A  from  Adam,  et .  al . : 

9.  Generate  key  deviation  measures. 

10.  Collect  technology  and  systems  data. 

11.  Edit  measures  into  productivity  ratios. 
Procedure  B  from  Sink  and  Tuttle: 

4.  Data  requirements  to  provide  information  in  the 
previous  Sink  and  Tuttle  step. 

Procedure  C  from  the  Office  of  Management  and  Budget: 

4.  Develop  quality  measures  of  indicators. 
Procedure  D  from  Christopher: 

3.  Determine  the  most  useful  measures  of  output. 

4.  Identify  and  measure  inputs. 

5.  Develop  appropriate  productivity  measures. 
Procedure  E  from  Van  Per  Ven: 

Phase  3:  Criteria  Development. 

Procedure  F  from  Siegel: 

6.  Design  of  measurement  system. 

7.  Installation  and  "debugging". 
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TABLE  26 

STEP  5:  ASSESS  THE  MEASURES 


Procedure  A  from  Adam,  et .  al . : 

12.  Clarify  edited  measures  and  revise  as  necessary 
Procedure  B  from  Sink  and  Tuttle: 

5.  The  data  to  information  transformation. 

Procedure  C  from  the  Office  of  Management  and  Budget: 

5.  Assess  quality  measures. 

Procedure  D  from  Christopher: 

6.  Report  measures  for  relevant  reporting  period 
and  monitor  trends  over  time. 

Procedure  E  from  Van  Per  Ven: 

Phase  4:  Evaluation  Design. 

Phase  5:  Evaluation  Implementation. 

Phase  6:  Data  Analysis,  Feedback,  and  Evaluation. 
Procedure  F  from  Siegel: 

8.  Instructions  for  operation  and  recommendation 
for  evaluation. 
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6.2.2  Summary  of  the  Interviews.  The  secondary  purpose 
for  this  thesis  was  to  determine  if  the  integrated  approach, 
based  on  the  literature,  could  be  grounded  in  private 
industry  practice  at  Pratt  &  Whitney.  Note  that  the 
information  presented  in  each  interview  is  one  person's  view 
of  how  the  measurement  development  process  took  place.  Even 
with  this  limitation,  the  information  is  quite  valuable. 

Not  only  was  this  information  derived  directly  from  the 
manager  of  the  department,  the  person  likely  to  have  the 
most  complete  knowledge  of  the  workings  of  the  department, 
it  was  derived  from  a  semi-structured  interview  designed 
specifically  to  determine  what  the  measurement  development 
procedure  was,  not  to  determine  if  the  approach  used  was 
right  or  wrong. 

Taking  each  step  of  the  integrated  approach  in  order. 

Step  1:  Establish  a  measurement  team,  is  the  first  step  to 
be  considered.  Only  two  of  the  twelve  department  managers 
interviewed  actually  had  a  measurement  team.  In  some  cases, 
the  managers  themselves  decided  what  the  measure  of 
performance  would  be.  In  other  cases,  the  measures  were 
imposed  by  the  company.  This  is  in  direct  conflict  wit.)  the 
unanimous  recommendations  contained  in  the  literature.  Each 
of  the  department  managers  that  did  not  form  a  measurement 
development  team  did  state  that  they  had  achieved  buy-in 
within  the  department  for  the  measures.  They  also  claimed 
that  the  measures  were  working.  Without  interviewing  the 
entire  department,  it  is  difficult  to  determine  whether  or 
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not  the  members  actually  felt  the  measures  worked.  From  the 
interview  information,  though,  it  appears  that  the  perceived 
key  to  success  is  achieving  buy-in  from  the  members  of  the 
department . 

Next,  consider  Step  2:  Obtain  a  clear  understanding  of 
the  organization.  Each  manager  felt  that  the  members  of 
their  department  had  a  clear  understanding  of  the 
organization.  Many  of  the  departnents  had  printed  mission 
statements  which  the  members  posted  in  a  visible  location. 
Each  of  the  department  managers  held  periodic  personal 
performance  reviews  with  each  employee  to  evaluate  their 
particular  piece  of  the  department's  work  effort.  From  the 
information  presented,  all  the  managers  followed  this  step. 

The  next  step  to  consider  is  Step  3:  Focus  on 
measurements  in  the  areas  of  key  importance.  Once  again, 
each  of  the  managers  interviewed  felt  that  measures  used  by 
their  department  were  focused  on  the  areas  of  key 
importance.  Since,  in  most  cases  the  managers  had  the  power 
to  change  the  measures,  the  very  fact  that  they  were  in  use 
lends  credence  to  the  manager's  opinion  that  the  measures, 
at  least  from  their  perspective,  were  effective. 

Step  4:  Classify  information  needs  is  the  next  step  to 
consider.  Each  of  the  managers  felt  that  the  measures 
provided  the  information  they  needed.  Since  the  managers 
also  stated  that  the  employees  within  the  department  bought 
into  the  measures,  the  managers'  perception  was  that  the 
employees  must  also  have  felt  that  the  measures  provided  at 
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least  some  necessary  information. 

Finally,  consider  Step  5:  Assess  the  measures.  All 
managers  accomplished  this  step.  The  managers  also  stated 
that  their  department's  measures  worked.  As  noted  earlier, 
most  of  the  managers  had  the  power  to  change  the  measures  if 
they  felt  a  change  was  necessary.  Since  the  measures  were 
in  use,  one  can  surmise  that  the  managers  must  feel  that 
they  are  of  more  benefit  than  any  of  the  available 
alternatives . 

The  interviews  are  summarized  in  Table  27.  Of  all  the 
interviews  conducted,  only  two  departments  provided  textbook 
examples  of  how  to  develop  departmental  measures  of 
performance.  Those  two  groups  were  the  Support  Equipment 
Group  and  the  J58  Engine  Group.  Thus,  to  say  that  the 
measurement  development  procedure  was  entirely  grounded  in 
the  interviews  with  the  departments  at  Pratt  &  Whitney  would 
be  incorrect.  In  most  cases  the  first  step  was  not  followed 
because  the  measures  were  provided  from  an  external  source 
to  the  department,  because  of  changeover  within  the 
department,  or  because  another  source  discovered  a  useful 
measure  without  the  aid  of  a  team.  However,  it  is  important 
to  note  that  the  rest  of  the  integrated  approach  was 
followed  to  maintain  appropriate  measures  of  performance. 

Each  department  also  seemed  to  use  measures  that  were 
suitable  for  its  own  function.  The  measures  they  developed 
provide  good  examples  for  a  department  starting  this 
process,  but  to  say  that  one  specific  measure  is  appropriate 
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TABLE  27 


THE  STEPS  OF  THE  INTEGRATED  APPROACH  AND  THE 
DEPARTMENTS  THAT  ADHERED  TO  THE  STEPS 


INTEGRATED  APPROACH 

DEPARTMENTS 

1.  Establish  a  measurement 
team. 

Support  Equipment  Group 

J58  Engine  Group 

2.  Obtain  a  clear 
understanding  of  the 
organization. 

All 

3.  Focus  on  measurements 
in  the  areas  of  key  importance. 

All 

4.  Classification  of 
information  needs. 

All 

5.  Assessment  of  measures 

All 
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for  a  number  of  different  departments  would  be  contradictory 
to  the  data  presented,  as  well  as  to  the  literature. 

6.2.3  Summary  Findings.  The  literature  reviewed 
provided  six  measurement  development  techniques  that  were 
integrated  into  a  single  model,  without  violating  any  of  the 
concepts  of  the  six  authors.  What  is  more,  the  integrated 
technique  seems  to  be  theoretically,  useful  whether  it  is  to 
be  used  to  develop  performance  measures,  productivity 
measures,  or  quality  measures. 

Ten  of  the  twelve  department  managers  interviewed  did  not 
follow  the  general  guidelines  developed  from  the  literature 
however,  when  developing  their  measures  of  performance.  The 
measures  they  used,  according  to  the  managers,  were  working, 
though.  There  may  be  several  reasons  for  the  supposition 
that  the  measures  worked.  First,  the  employees  may  believe 
in  the  measures  and  therefore  provide  their  support. 
Management  comments  tend  to  support  this  view.  Second, 
those  who  developed  the  measures  may  know  a  great  deal  about 
the  department  and  therefore  know  what  type  of  measure  works 
or  does  not  work.  In  either  case,  the  reasons  behind  why 
the  measures  work  is  a  subject  for  further  study  that  may 
lead  to  greater  understanding  of  Step  One  of  the  integrated 
approach.  Finally,  it  is  also  important  to  note  that  all  of 
the  other  steps  in  the  integrated  approach  were  reported  to 
be  in  use. 
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6.3  Recommendations  for  Further  Research. 


6.3.1  System  Program  Office  Test.  The  next  logical  step 
in  this  process  is  to  tsst  the  integrated  approach  in  a  SPO 
environment,  focusing  first  on  the  smallest  units  within  the 
SPO  that  have  a  specific  mission.  The  researcher  should 
identify  the  test  subjects,  work  with  them  to  employ  the 
steps  involved  in  the  integrated  approach,  and  then  give  the 
departments  time  to  use  the  derived  measures  and  get  a  feel 
for  their  effectiveness.  Only  after  a  period  of  use  could 
the  department  supervisors  determine  whether  the  measurement 
developing  method  worked,  actually  advancing  the  interests 
of  the  department. 

Useful  information  would  be  drawn  if  some  of  the 
departments  chosen  for  the  test  performed  similar  functions 
as  the  J58  Engine  Program  Office  or  the  Support  Equipment 
Group.  These  are  the  two  departments  within  Pratt  &  Whitney 
that  used  a  measurement  development  approach  closely  related 
to  the  integrated  approach.  The  examples  of  measures  from 
these  two  departments  may  prove  useful  for  generating 
initial  ideas  for  the  SPO  departments.  Also,  the  duties 
performed  in  these  two  departments  very  closely  resemble  the 
duties  performed  in  a  program  office  or  support  equipment 
office  in  a  SPO  as  far  as  who  their  customers  and  suppliers 
are  and  the  tasks  that  are  accomplished. 

6.3.2  Other  Areas  of  Further  Research.  Other  areas  of 
further  research  include  the  following: 
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1.  Determine  if  the  measures  developed  in  an 
Air  Force  SPO,  using  the  integrated  approach, 
are  different  than  those  used  by  the  Pratt  & 
Whitney  departments  that  had  measures  thrust 
upon  them. 

2.  Determine  if  the  departments  that  did  not 
have  a  measurement  system  tried  the 
integrated  approach  recommended  here,  if  they 
tried  some  other  approach,  or  if  they  have 
not  tried  measurement  at  all.  If  they  tried 
this  approach,  the  reasons  for  its  failure 
would  be  of  interest.  (It  should  be  noted 
that  the  only  departments  considered  for 
interviewing  in  the  current  study  were  those 
that  already  had  an  established  measurement 
system) . 

3.  Find  out  what  development  procedures 
businesses  other  than  Department  of  Defense 
contractors  use.  This  could  reveal  new 
knowledge,  or  provide  further  proof  as  to  the 
external  validity  of  the  integrated  approach 
presented  here. 

4.  Work  with  a  business  in  private  industry 
as  a  facilitator  to  implement  the  integrated 
approach  in  much  the  same  fashion  as  would  be 
done  in  the  SPO.  THis  would  also  add  to  the 
reliability  of  the  integrated  approach  and. 
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if  successful,  would  expand  the  approach 
application . 

5.  Determine  how  Japanese  industries  handle 
this  problem.  Since  departmental  measurement 
is  an  integral  part  of  the  quality  movement, 
and  since  the  Japanese  are  the  front-runners 
in  the  movement,  it  could  prove  beneficial  to 
learn  what  they  are  really  doing  in  this 
field. 

6.5  Conclusion. 

The  primary  intent  of  this  research  was  to  determine 
whether  a  measurement  development  procedure,  designed  for  a 
white-collar  department,  could  be  found  in  current 
literature.  This  was  accomplished.  As  evidenced  by  the 
information  presented  in  Chapter  4  and  summarized  in  Chapter 
6,  an  integrated  measurement  development  procedure  was 
constructed  based  on  the  analysis  of  the  literature.  The 
secondary  intent  of  the  research  was  to  determine  whether 
this  measurement  development  procedure  could  be  grounded  in 
private  industry  practice  at  Pract  &  WHitney.  The  study 
found  that,  although  most  of  the  steps  in  the  integrated 
approach  were  followed,  the  first  step,  establish  a 
measurement  team,  was  not.  Only  two  of  the  twelve 
department  managers  interviewed  had  followed  measurement 
development  procedures  similar  to  the  integrated  approach. 
The  answer  to  the  first  question  fulfills  ASD's  need  for  a 
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departmental  performance  measurement  development  procedure 
that  they  can  use  to  begin  to  further  their  Total  Quality 
efforts.  The  second  answer  provided  examples  of  white- 
collar  measures,  grounded  most  of  the  integrated  procedure, 
and  provided  numerous  ideas  for  further  research. 
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Appendix  A:  Sample  Letter 


Appendix  A  is  an  example  of  the  letter  sent  to  Pratt  & 
Whitney  to  introduce  the  subject  to  the  person  acting  as  the 
interview  coordinator  and  to  set  the  parameters  for  the 
types  of  managers  needed  for  the  interviews. 

To  whom  it  may  concern: 

1.  I  would  like  to  interview  department  managers  for  the 
purpose  of  determining  what  types  of  department  performance 
measurement  techniques  they  use  and  how  these  techniques 
were  developed. 

2.  For  the  purpose  of  our  discussion,  we  will  use  the 
definition  that  a  department  is  the  lowest  (on  the 
organization  chart)  organized  group  of  people  working  under 
the  same  mission  and  toward  the  same  goals.  An  example 
would  be  a  program  office  within  the  Air  Force,  or  a 
financial  branch  assigned  to  a  particular  project.  The 
managers  must  be  from  areas  specializing  in  white-collar 
efforts,  that  is,  there  is  no  hardware  production  as  an 
output  of  their  department. 

3.  'his  research  is  to  support  a  thesis  on  the  topic  of 
departmental  measurement,  specifically,  how  to  develop 
useful  measures  of  performance  within  a  white-collar 
environment.  I  am  currently  enrolled  in  the  graduate  degree 
program  at  the  Air  Force  Institute  of  Technology.  These 
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interviews  will  provide  data  for  the  thesis  research  effort. 
The  initial  interview  questions  are  attached.  What  I  am 
looking  for  is  a  trend  within  your  company  on  how  to  develop 
measures  of  performance  within  a  department. 

4.  After  completion  of  the  thesis,  I  would  be  happy  to 
provide  a  copy  of  it  to  anyone  within  the  your  company 
participating  in  the  research,  should  they  so  desire. 

5.  I  will  call  you  to  set  up  the  interview  schedules.  If 
you  have  any  questions  prior  to  that  time,  please  call  me  at 
513-255-4437  (work)  or  at  513-438-8025  (home).  Thank  you. 

Sincerely,  1  atch: 

Interview  Questions 
(see  Appendix  B) 


KIRK  H.  RUMSEY,  Capt ,  USAF 


i 


« 
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Appendix  B:  Interview  Questionnaire 


Introduction 

Who  I  am,  where  I'm  from. 

This  effort  is  for  a  thesis  study. 

I  am  looking  for  your  opinions.  There  is  no 
attribution,  no  wrong  answers. 

Do  you  want  a  copy  of  the  results? 

QUESTIONNAIRE 

1.  Can  you  tell  me  about  the  program  of  which  you  are  in 
charge? 

What  types  of  duties  does  your  department  perform? 

How  many  people  work  for  you? 

2.  What  are  your  department’s  primary  products/services? 

3.  Who  are  the  customers  for  each  product? 

4.  How  do  you  measure  the  performance  of  your  department? 

Do  you  have  a  single  measure  of  performance,  or 
many? 

What,  specifically,  do  you  measure? 

Examples? 

Who  receives  these  performance  measurement  indicators? 
What  do  you  do  with  these  measures? 

5.  How  did  you  arrive  at  these  measures? 

Were  they  dictated  from  the  upper  level? 

Did  you  work  with  the  members  of  your  department? 

Did  you  work  with  your  customer(s)? 

Do  they  work? 

Are  they  reviewed? 

Would  you  change  them  if  you  could?  If  so,  how? 
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